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Abstract. Rankin-Cohen bidiffcrcntial operators are the projectors onto irre- 
ducible summands in the decomposition of the tensor product of two represen- 
tations of SX(2,R). We consider the general problem to find explicit formulae 
for such projectors in the setting of multiplicity-free branching laws for reductive 
symmetric pairs. 

For this purpose we develop a new method (F-method) based on an algebraic 
Fourier transform for generalized Verma modules, which enables us to characterize 
those projectors by means of certain systems of partial differential equations of 
second order. 

As an application of the F-method we give a solution to the initial problem by 
constructing equivariant holomorphic differential operators explicitly for all sym- 
metric pairs of split rank one and reveal an intrinsic reason why the coefficients of 
Jacobi polynomials appear in these operators including the classical Rankin-Cohen 
brackets as a special case. 
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1. Introduction 

What kind of differential operators do preserve modularity? This question was 
studied by R. A. Rankin |Ra56j and later investigated by H. Cohen [C75j in the 
framework of bidifferential operators. These authors introduced a family of operators 
transforming a given pair of modular forms into another modular form of a higher 
weight. Let fx and f 2 be holomorphic modular forms for a given arithmetic subgroup 
of SL(2,M) of weight k\ and k 2 , respectively. The bidifferential operator, referred to 
as the Rankin-Cohen bracket of degree a and defined by 

(i-i) Kci lM {hj 2 ){z):=±{-iy(^ fcl 7 _1 )( 

yields a new holomorphic modular form of weight k\ + k 2 + 2a. 

These operators have attracted considerable attention in recent years particularly 
because of their various applications in 

- theory of modular and quasimodular forms (special values of L-functions, 
the Ramanujan and Chazy differential equations, van der Pol and Niebur 
equalities) [CT7PT1 IMR09IIZ94], 

- covariant quantization [BTY071 ICMZ971 ICM041 IDP071 lP08l IUU96] , 

- ring structures on representations spaces )DP07t IZ94j . 

Existing methods for the SL(2, R)-case. A prototype of Rankin-Cohen brack- 
ets was already found by P. Gordan and S. Guldenfinger |Gol887| IGul886] in the 
19th century by using recursion relations for invariant binary forms and the Cay- 
ley processes. For explicit constructions of such equivariant bidifferential operators 
several different methods have been developed: 

- Recurrence relations [U75l IBIM IHT921 IP121 [Z94] 

- Taylor expansions of Jacobi forms [EZ85, I KQ12t lKu75j . 

- Reproducing kernels for Hilbert spaces [PZQH IUU96t IZhlOj . 

- Dual pair correspondence |B06t IEI98] . 

We propose yet another method based on branching laws and the algebraic Fourier 
transform of generalized Verma modules and apply it to a family of more general 
semisimple Lie groups beyond the SL(2,M) case (see Table |4TT|) . 

Branching laws for symmetric pairs. By branching law we mean the decom- 
position of an irreducible representation ir of a group G when restricted to a given 
subgroup G'. An important and fruitful source of examples is provided by pairs of 
groups (G, G') such that G' is the fixed point group of an involutive automorphism 
a of G, called symmetric pairs. 

The decomposition of tensor product is a special case of branching laws with 
respect to symmetric pairs (G,G r ). Indeed, if G = G\ x G\ and a is an involutive 



r(*)irT(*) s 
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automorphism of G given by a{x,y) = (y,x), then G' - G\ and the restriction of the 
outer tensor product tti M tx 2 to the subgroup G' is nothing but the tensor product 
7Ti ® 7T2 of two representations tti and 7r2 of G*i. The Littlewood-Richardson rule 
for finite dimensional representations is another classical example of branching laws 
with respect to the symmetric pair (GL(p + q, C), GL(p, C) x GL(q,C)). 

Rankin— Cohen operators as intertwining operators. From the view point of 
representation theory the Rankin-Cohen operators are the projectors to irreducible 
components in the tensor product of two holomorphic discrete series representations 
of SL(2,M). More precisely, the operator (11.11) is the projector to the summand 
^k 1 +k 2 +2a in the following abstract branching law [M08CH lRe79j : 

(1-2) 7T fel <g> 7C k2 ~ 2^ 71"fe 1+ fc 2 +2a- 

aeN 

The subject of this paper is to develop a method to find explicit projectors onto 
irreducible components of branching laws in a broader setting of symmetric pairs. A 
reasonable general assumption to work with would be the multiplicity-free decompo- 
sitions where the projectors are unique up to scalars. This is the case, for instance, 
if the representation n is any highest weight module of scalar type (or equivalently n 
is realized in the space of holomorphic sections of a homogeneous holomorphic line 
bundle over a bounded symmetric domain) and (G,G f ) is any symmetric pair (see 
|K08[ IK 12] for the multiplicity- free theorems). 

Geometric setting. We shall work in the following geometric setting. Let Vx X 
be a homogeneous vector bundle of a Lie group G and Wy -» Y a homogeneous 
vector bundle of G'. Then we have a natural representation n of G on the space 
F(X,Vx) of sections on X, and similarly that of G' on r(Y,Wy). Assume G' is a 
subgroup of G. Our main concern is with the following question: 

Question 1. Find explicit G' -intertwining operators from V(X, Vx) to T(Y, Wy). 

In the case where X is a Hermitian symmetric domain, Y a subsymmetric domain, 
G'cG are the groups of biholomorphic transformations of / : Y X, respectively, 
and Vx is any line bundle C\ with sufficiently positive parameter A, we prove: 

Theorem A. Any continuous G' -homomorphism from 0(X,C\) to 0(Y,W) is given 
by a unique, up to a scalar, differential operator. 

See Theorem 14.31 for precise statement. Theorem lAl includes the tensor product 
case, namely, G - G' x G' and X ~ Y x Y as a special case. In contrast to Theo- 
rem A, we notice that in general, the intertwining operators between two unitary 
representations of real reductive Lie groups G' c G are given by integro-differential 
operators in geometric models. Among them, equivariant differential operators are 
very special (e.g. |KS7R IKS] ). 
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We observe that the pullback /* of sections is obviously G'-intertwining if Wy - 
f*Vx- Finding all bundles Wy for which such nontrivial intertwining operators exist 
is a part of the above question, which reduces to abstract branching problems (see 
Theorem 12.7ft . However, giving explicit formulae of intertwining operators is more 
involved even when abstract branching laws are known, as we may note by comparing 
( II. ip with (II. 2p in the SX(2,IR) case. In fact, to answer Question 1 one has to know 
a finer structure of branching laws, namely, the precise places where the irreducible 
summands are located in the whole representation space. 

F-method. To answer Question [TJ we propose a method, that we call F-method, 
which consists of two stages: 

(a) Transfer the initial question to an algebraic problem of understanding a fine 
structure of branching laws for induced modules of enveloping algebras. 

(b) Characterize projectors by means of certain systems of partial differential 
equations. 

Concerning the stage (a) we first prepare a rigorous notion of 'differential operators' 
between two manifolds X and Y with morphism / : Y -> X and then we prove in 
Theorem 12.71 a one-to-one correspondence between G'-equivariant differential oper- 
ators and g'-homomorphisms of induced modules (algebraic branching laws). This 
generalizes a well-known result in the case where G = G' and X = Y are the same 
flag variety ([HJ82J). The multiplicity-freeness of algebraic branching laws for gen- 
eralized Verma modules guarantees the uniqueness of the projectors in Question 1 
when X and Y are flag varieties. 

We proceed to Stage (b) under the assumption that X is defined by a parabolic 
subalgebra with an abelian nilradical. We then characterize g'-homomorphisms of 
generalized q- Verma modules by means of certain systems of partial differential equa- 
tions. It should be noted that the system is of second order although the resulting 
covariant differential operators may be of any higher order. The characterization is 
performed by applying an algebraic Fourier transform (see Definition 13. ip . 

The detailed recipe of the F-method is described in Section 1331 relying on Theorem 
13.91 and Proposition 13.111 

In Sections and O we give an answer to Question [T] for all symmetric pairs 
(G, G') of split rank one inducing a holomorphic embedding Y ^ X (see Table Id]) . 
We remark that the split rank one condition does not force the rank of G/G' to be 
equal to one (see Table [TTT1 (1), (5) below). 

Normal derivatives and Jacobi— type differential operators. In representation 
theory, taking normal derivatives with respect to an equivariant embedding Y <-> X 
is sometimes a useful tool to find abstract branching laws for representations that 
are realized on X (see S. Martens [M75| and H. P. Jakobsen and M. Vergne [JV79j ). 
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This method works well for filtered modules, but normal derivatives themselves 
are not always intertwining operators as it is already the case for the tensor product 
of two holomorphic discrete series of SL(2,M) where the only equivariant operators 
are Rankin-Cohen bidifferential operators. 

By using the F-method, we find a necessary and sufficient condition for normal 
derivatives to become intertwiners and give a complete classification in the case where 
the symmetric pair is of split rank one (see Theorem 15. ip . Namely, it happens for 
the equivariant embeddings of associated flag varieties given exactly in the second 
column of the following table and fails for those in the first column. 



(1) F n C ^ P n C x P«C (4) Gr p _i(CP + 9) ^ Gr p (O> + <0 

(2) LGr(C 2n - 2 ) x LGr(C 2 ) ^ LGr(C 2n ) (5) ' P"C ^ Q 2 "C 

(3) Q"C ^ Q" +1 C (6) IGr^C 2 "" 2 ) - IGr n (C 2n ) 

Table 1.1. Equivariant embeddings of flag varieties 



Here Gr p (C n ) is the Grassmanian of p-planes in C n , Q m C := {z e P m+1 C : Zq + ■■■ + 
z m+i = 0} * s the complex quadric, and IGr n (C 2n ) := {V c C 2 ™ : dimV^ = n, Q\ v = 0} is 
the Grassmanian of isotropic subspaces of C 2n equipped with a quadratic form Q, and 
LGr n (C 2n ) := {V c C 2 " : dim"!/ = n, u\yxv - 0} is the Grassmanian of Lagrangian 
subspaces of C 2n equipped with a symplectic form ui. 

All the differential intertwining operators appearing in the geometric framework 
of the second row of the Table 11.11 are not given by normal derivatives but are built 
on Jacobi polynomials in one variable. More precisely, let P^ ,l3 (x) be the Jacobi 
polynomial, and C"(x) the Gegenbauer polynomial. We inflate them to polynomials 
of two variables by 

/^(*,y):= 1^(2^ + 1) and C°(x,y) := x^Cf (~?=j ■ 

Then we obtain explicit formulae for equivariant differential operators correspond- 
ingly to (1), (2), and (3) in Table ED 

Theorem B. In that follows, C\ stands for a homogeneous holomorphic line bundle, 
and yV% a homogeneous vector bundle with typical fiber S a (C m ) (m = n in (1); = n-1 
in (2)) with parameter A (see Lemma \5. 3\ for details). 

(1) For the symmetric pair (U(n, 1) x U(n, 1), U(n, 1)) the differential operator 

f) ._ p-\'+n,\'+\"-2n-2a+l I \* ® V„ ® \ 
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is an intertwining operator from 0(Y, \^)®0(Y, £(a^,a^)) ^° ®(Xi ^(a'+a" a'+a"))' 
where A' 1; A", \' 2 , A£ eZ, X' = X[ - X' 2 , X" = X'{ - X'J,, and a e N. 

(2) For the symmetric pair (Sp(n,M.),Sp(n -1,R) x 5^(1, R)) the differential op- 
erator 




is an intertwining operator from 0(X,C\) to 0(Y, W^) 7 where X e Z, oeN. 
(3) For the symmetric pair (SO(n,2) , SO(n - 1,2)) the differential operator 

Dx^Y,a'-=C a 2 ^-A C n-i,-^— j 

is an intertwining operator from 0(X,£\) to 0(Y,£\ +a ), where A e Z and a e N. 

See Theorems I6.14[ 16. 9[ and 16.11 for precise statements, respectively. The above 
operators exhaust all intertwining operators for generic parameter A (or (A', A") in 
(1)), see Theorem 14.31 

The first statement corresponds to the decomposition of the tensor product, and 
gives rise to the classical Rankin-Cohen brackets in the case where n = 1. The 
third one may be thought of as a 'holomorphic version' of the conformally covariant 
operators constructed recently by A. Juhl [J09] by a completely different method in 
the setting S n S n+1 , with equivariant actions of the pair of groups SO(n + 1,1) c 
SO(n + 2,1), respectively. 

Last but not least, we expect that the F-method could be a useful tool in other 
settings, for example, in constructing covariant operators in parabolic geometries (cf. 
K0SS]), and also in developing a theory of equivariant differential operators built 
on orthogonal multivariable polynomials associated to root systems. 

2. Differential intertwining operators 

In this section we discuss equivariant differential operators between homogeneous 
vector bundles in a more general setting than usual, namely, over different base spaces 
admitting a morphism. In this generality, we establish a natural bijection between 
such differential operators and certain Lie algebra homomorphisms, see Theorem 12.71 

2.1. Pullback of differential operators. We understand the notion of differential 
operators between two vector bundles in the usual sense when the bundles are defined 
over the same base space. We extend this terminology in a more general setting, 
where there exists a morphism between base spaces. 

Definition 2.1. Let V -> X and W -*■ Y be two vector bundles and p ■ Y X a 
smooth map. We say that a linear map T : C°°(X, V) ->■ C°°(Y, W) is a differential 
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operator if T is a local operator in the sense that 

(2.1) Supp(Tf) cp-^Supp/), forany/ e C°°(X,V). 

We write Diff (Vx, VVy) for the vector space of such differential operators from C°°(X, V) 
toC°°(Y,>V). 

Since any smooth map is factorized into the composition of a submersion and an 
embedding, the following example describes the general situation. 

Example 2.2. (1) Let p : Y -» X be a submersion. Choose an atlas of local 
coordinates {(xi,Zj)} on Y in such a way that {xi} form an atlas on X. 
Then, every T e Diff(Vx, Wy) is locally of the form 

d\ a \ 

where h a (x,z) are Hom(V, W)-valued smooth functions on Y. 
(2) Let i :Y X be an embedding. Choose an atlas of local coordinates {(yi, Zj)} 
on X in such a way that {yi} form an atlas on Y . Then, every T e Diff (Vx, Wy) 
is locally of the form 

L 9a ^dv a dz^ 

where g a ,p(y) ar ^ Hom(V, W)-valued smooth functions on Y. 

Let Q x '■= I A top T*(X)\ be the bundle of densities, and denote by V* the dualizing 
bundle V v <B> fix- By the L. Schwartz kernel theorem, any continuous linear map 
T : C°° (X, V) -> C°° (Y, W) is given as 

f"(K T (x,y)J(x)), 

for some distribution Kt e T>'{X x Y, V* H W) such that the second projection pr 2 : 
X x y -> Y is proper on the support of Kt- 

Lemma 2.3. Letp ■ Y -> X be a smooth map. A continuous operator T : C°°(X, V) -» 
C°°(Y, W) a differential operator in the sense of Definition \2.1\ if and only if 
SuppK T c A(Y), where A(Y):={(p(y),y):yeY}cXxY. 

Proof. Suppose Suppi^ c A(Y). By the structural theory of distributions sup- 
ported on a submanifold |S66l Chapter III, Theoreme XXXVII] to Ay c X x Y, we 
have (Kt, /) e ©'(Y, W) is locally given as 

(2-2) EM*)|^(p(v)), 
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for every / € C°°(X, V) and some h a (y) e V'(Y,V* ® W). Thus T is a differential 
operator in the sense of Definition 12.11 

Conversely, by the definition of the kernel distribution Kt, for any (x ,y ) e 
Suppi^r and for any neighborhood S of x Q in X there exists / € C°°(X, V) such 
that Supp/ c S and (x ,y ) e Supp/ x SuppT/. If T is a differential operator then 
by (M 

(x , 2/ D ) e Supp / x p- 1 (Supp /) . 

Since S is an arbitrary neighborhood of x Q , then necessarily (x ,y ) e A(Y) and 
therefore Supp K T cA(Y). □ 

Next, suppose the two vector bundles V X and W -> K to be equivariant with 
respect to a given Lie group G. Then we have natural actions on the spaces C°°(X, V) 
and C°°(Y, W) by translations. We set 

Diff G (Vx,>Vy) :=Diff(Vx,Wy)nHom G (C7 00 (X,V),C 00 (y,>V)). 

Example 2.4. // X and V are 6o£/i Euclidean vector spaces and if G contains the 
subgroup of translations of Y then Di&a(Vx, Wy) as a subspace of the space of dif- 
ferential operators with constant coefficients. 

2.2. Induced modules. Let g be a Lie algebra over C, and U(g) its universal 
enveloping algebra. Let h be a Lie subalgebra of g. 

Definition 2.5. For an f)-module V we define the induced [/(g)-module as 

indg(F) := [/(g) ® P(W V. 

If h is a Borel subalgebra and dimy = 1, then ind®^) is the standard Verma module. 

For further purposes we formulate the following statement in terms of the contra- 
gredient representation V w . Let h' be another Lie subalgebra of g. 

Proposition 2.6. For finite dimensional -module W we have 

(1) Hom g (indS,(W v ),indS(y v )) ~ Hom h ,(V^ v , ind^V^)). 

(2) Ifh'th, then Hom^V^ind^V^)) = {0}. 

Proof. The first statement is due to the functoriality of the tensor product. 

For the second statement it suffices to treat the case where h' is one- dimensional. 
The assumption t)' ^ f) implies that there is a direct sum decomposition of vector 
spaces: 

= f)' + q + f), 

for some subspace q in g. Then, by the Poincare-Birkhoff-Witt theorem we have an 
isomorphism of (/-modules: 

ind«(V v )^ £/({,') ® c C/'(q)® c V\ 
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where U'(q) := C-span{XyXe ■ X 1 ,. . . ,X e e q, £ e N}. In particular, ind?(V v ) is a 
free U (f/)-module. Hence no finite dimensional f)'-submodule exists in indj|(V v ). □ 

2.3. Differential operators between homogeneous vector bundles on differ- 
ent base spaces. Let G be a real Lie group, fl(K) := Lie(G) and g := fl(M) ® C. 
Analogous notations will be applied to other Lie groups. 

Consider two actions dR and dL of the universal enveloping algebra U(q) on 
the space G°°(G) of smooth complex- valued functions on G induced by the regular 
representation L x R of G x G on G°°(G): 



(2.3) (dfl(X)/)(x) := | 



f(xe tx ), and (dL(X)f)(x) := ^> 



/(e- iX x). 

t=o 



Let i7 be a closed subgroup of G. Given a finite dimensional representation 1/ of 
H we define the homogeneous vector bundle Vx = V ■= G x H V over X := G/H. The 
space of smooth sections G°°(X, V) can be seen as a subspace of G°°(G) ® 1/: 

G°°(X,V) = r(G,F) H cr(G)0l/. 

The right differentiation given by (12.31) 

G°°(G) x [/(g) ^ G°°(G), (f,u)~dR(u)f 

together with the canonical coupling V x V v ->• C induces a well-defined diagram of 
maps: 

C°°(G)®V xU(q)® c V v — ► G°°(G) 
t j. || 

C M (I,V) xind3(V v ) — * G°°(G), 

because G°°(X, V) - G°°(G, V) H . In turn, we get the following g-homomorphism: 

(2.4) ind*(y v ) ^Hom G (G°°(X,V),G°°(G)), 

where G°°(G) is regarded as G x g-module via L x dR. 

Consider another connected closed Lie subgroup H' . Given a finite dimensional 
representation W of H' we form a homogeneous vector bundle Wz = W-=G x H , W 
over Z :- G/H'. Taking the tensor product of (12.41) with W, we get a morphism: 

Hom c (iy v ,indS(y v ))^Hom G (G 00 (X,V),G 00 (G,H/)). 

Taking f) '-invariants, we have then a morphism 

(2.5) Hom„,(H/ v ,indS(y v )) ^ Hom G (G~(X, V),G°°(Z,}V)), p ~ 

We note that the map (12.51) is injective. 

Take any subgroup G' containing H' and form a homogeneous vector bundle Wy '■= 
G'xh>W over Y = G'/H'. Then, the vector bundle Wy is isomorphic to the restriction 
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Wz\y of the vector bundle Wz to the submanifold Y of the base space Z. Let 
Rz^y ■ C°°(Z,Wz) -> C°°(F,>Vy) be the restriction map of sections. We set 

(2.6) ZW(¥>):=iW>iV 

The next theorem describes explicitly the image Dx^y which, according to Propo- 
sition 12. 6[ is non-trivial only when H' c H, that is, when the following diagram 
exists: 

Z = G/H' 




Y = G'/H' X = G/H 

Theorem 2.7. Let G be a Lie group, H and H' are closed connected subgroups such 
that H' c H . Consider two finite dimensional representations V and W of H and H', 
respectively. Then, for any G' containing H' the map Dx^y establishes a bijection: 

(2.7) Dx^y ■■ Hom^(^ v ,ind^(y v )) ^ Diff G ,(V x , Wy). 

Remark 2.8. (1) By the functoriality of the tensor product, the bijection (12. 7p 
may be restated as 

fl2ZD ' D X ^Y ■■ Hom , (indj ) , ind« ( V v ) ) Diff G , ( V x , Wy ) . 

(2) Theorem 12. 71 is well-known when X = Y, i.e. G' = G and H' = H, in particular 
in the setting of complex flag varieties, see e.g. |HJ82t IKs74] . 

(3) We shall consider the case where H' = H n G' in later applications, however, 
the setting of Theorem 12.71 covers also the cases where the natural morphism 
Y -> X is not injective, i.e. where H' ^ H n G' . 

(4) The left-hand side of (12. 7p does not depend on the choice of G' . This fact is 
reflected by the commutativity of the following diagram. 

(2.8) Hom^Vnd^)) — Diff G (V x , W z ) 




Rz-,y 



Diff G ,(V X ,>Vy) 

Before giving a proof of Theorem 12 .71 we state its variants, namely, for disconnected 
subgroups (Corollary 12. 9j) and for the holomorphic case (Proposition 12 . 1 0|) . 

In dealing with a representation V of a disconnected subgroup H, we notice that 
the diagonal if-action on U (g) <8>c ^ v defines a representation of H on ind[j(y v ) and 
thus ind?(y v ) is endowed with a (q, if)-module structure. 
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Corollary 2.9. Let H' c H be (possibly disconnected) closed subgroups of G with Lie 
algebras h' c h, respectively. Suppose V and W are finite dimensional representations 
of H and H' , respectively. Let G' be any subgroup containing H', and Vx '■= G x-hV 
and Wy '■= G' x H ' W be the corresponding homogeneous vector bundles. Then, there 
exists the following natural bijection: 

D x ^y : Hom^(W v ,ind3(V v )) — Diff G , (V X ,W Y ) , 

or equivalently, 

Dx^y : Rom {grtHr) (md{ r (W^),mdl(V v )) — Diff G , (V X ,W Y ) , 

Proposition 2.10. Theorem \2. 7| remains valid for complex Lie groups and equivari- 
ant holomorphic differential operators. 

The proofs of Corollary 12.91 and Proposition 12 . 1 01 are parallel to the one of Theorem 
12.71 which is based on the explicit construction of the bijection Dx^y- 

2.4. Proof of Theorem 12.71 The proof for the surjectivity of Dx^y reduces to a 
realization of the induced J7(g)-modules indjj(V) in the space of distributions. We 
begin with some notations. 

Let C 2p denote the one- dimensional representation of H defined by 

h^\dest(AdG /H (h):a/i)^a/i))\-K 

The bundle of densities VL g /h is given as a G-equivariant line bundle, 

-Gx H \ det -1 Adc/H \-Gx H C 2p . 

Given an iZ-module V, we define a 'twist' of the contragredient representation by 

V 2 v p ■= V v ® | det - 1 Adc/H I - V v ® C 2p . 

Then the dualizing bundle is given, as a homogeneous vector bundle, by: 

(2.9) V* e V 2 v p := V v 9 Q g /h - G x H V£. 

In what follows T>'(X, V*) (respectively, £'(X, V*)) denotes the space of ^"-valued 
distributions (respectively, those with compact support). We shall regard distribu- 
tions as generalized functions a la Gelfand rather than continuous linear forms on 
C™(X,V) (respectively, C°°(X,V)), and write f x u for the pairing of u e £'(X,Q X ) 
and the constant function l x on X. For a compact subset S of X, we write 
T>' S (X, V*) = £' S (X,V*) for the space of distributions supported on S. 

For a homogeneous vector bundle V, we shall use the notation V 2p rather than V*. 
Let o = eH e X. Define a vector valued Dirac 5-f unction 
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by 

(2.10) (/, 5 ® v^) := (/(e), u v ), for / e C7°°(X, V) * C°°(C7, V)" 
The following lemma is standard: 

Lemma 2.11. 

(1) The G-invariant functional cu f x co induces a G-invariant bilinear map 

(2.11) C°°(X,V)x£'(X,V% p )^C. 

(2) Let S be a closed subset of X and U an open neighborhood of S in X. Then, 
by restriction, the above map gives rise to a g-invariant pairing: 

c-{uy)xE' s {x^ 2p )^c. 

(3) The map 

(2.12) i n d;(yv)^ o] (Xj V v }) u g, v v ^ dL(u) (5 uV) ; 

is a Q-isomorphism. 

Now let us consider the setting of Theorem 12.71 where we have a G'-equivariant 
(but not necessarily injective) morphism from Y = G'/H' to X = G/H. 

Lemma 2.12. Suppose that G' is a subgroup of G. Then the multiplication map 

m:GxG'^G, M)^{gf)- X gi 
induces the isomorphism: 

m* : (V(X, V 2 v p ) ® W) A(H/) V\X x Y, V 2 v p m >V) A(G,) . 

Proo/. The image of the coproduct m* : V'(G) -> D'(GxG') is W(GxG') A ( G '\ where 
C acts diagonally from the left. Thus, considering the remaining GxG' action from 
the right, we take H x H'- invariants with respect to the diagonal action in 

V'{G) ® V 2 \ ®W ^ V\G x G') A(G,) ® V 2 v p ® W, 

and therefore we get Lemma. □ 

Lemma 2.13. There is a natural bisection: 

Diff G ,(Vx,W Y ) ^> (V' [o] (X,V x )® W)^ H '\ T»(m*)-\K T ). 

Proof. Let T e Differ (Vx, VVy). By Lemma 12.31 the distribution kernel is sup- 
ported on the diagonal set A(Y) cIxY. Via the bijection m* given in Lemma \2. 121 
we thus have 

Supp((m*)- 1 K T ) c { }. 
Conversely, take any element F e (V^(X, Vx)®W) A ( H '\ then m*(F) is supported 
on the diagonal set A(Y) c X xY. Since it is invariant under the diagonal action of 
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G', all the distributions h a in (12.21) are smooth. Therefore m*(F) defines a differential 
operator from C°°(X, V) to C°°{Y, W). □ 

Proof of Theorem \2. 7\ Taking the tensor product of each term in (I2.12p with the 
finite dimensional representation W of H', we get a bijection between the subspaces 
of i) '-invariants: 

Hom (l ,(Vr,ind^ v )) (V' [o] (X, V x ) ® W)^ H '\ 

Composing this with the bijection in Lemma 12.131 we obtain a bijection from 
Hom^(VF v , ind||(V" v )) to Diff G '(Vx, Wy), which is by construction nothing but D x ^y 
in Theorem 12.71 □ 

To achieve our goal to describe intertwining operators between discretely decom- 
posable generalized Verma modules we shall now make the correspondence if h> 
Dx^y(p) explicit. 

3. F-METHOD 

In this section we develop the F-method in details. The recipe is given in Section 
13.51 relying on the algebraic Fourier transform of generalized Verma modules (The- 
orem [33] and Proposition 13 .11]) . Some useful lemmas for actual computations for 
vector-bundle valued differential operators are collected in Section 13.61 

3.1. Weyl algebra and algebraic Fourier transform. Let E be a vector space 
over C. The Weyl algebra T>(E) is the ring of holomorphic differential operators on 
E with polynomial coefficients. 

Definition 3.1. We define the algebraic Fourier transform as an isomorphism of 
two Weyl algebras on E and its dual space E y : 

V(E) ^V{E y ), T»f, 

induced by 

T d , „ 

— :=-<,-, Zj :=—, l<j<n = d^E. 

where (zi, . . . , z n ) are coordinates on E and (Ci, . . . , ( n ) are coordinates on E v . 

Remark 3.2. Definition 13. II does not depend on the choice of coordinates. 

The natural action of the general linear group GL(E) on E yields a representation 
of GL(E) on the ring Pol(-E) of polynomials of E. Taking its differential, we get a 
Lie algebra homomorphism End (22) -^T>(E). In the coordinates this homomorphism 
amounts to 

(3.1) End(2?) - V(E), A » -WAd z e - A 
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Likewise, for the contragredient representation on E v we have 

(3.2) End(£)-D(£ v ), A ~ \Ad c ee £^0 A 

Further, a representation ct of on E gives rise to Lie algebra homomorphisms 
$> a :Q^T>(E) and # CT v : ^ X>(£ v ). 

Lemma 3.3. T/ie algebraic Fourier transform T T relates and as follows: 

ty a = ^o-v + Trace o a. 

Proof. By (13.11) and (13.21) the computation of the difference of and reduces 
to the commutation relations 

d d 

in the Weyl algebra T>(E W ). It yields Trace o a. □ 

For actual computations we need in the latter part of this work it is convenient 
to give another interpretation of the algebraic Fourier transform using specific real 
forms of E. 

Definition 3.4. Fix a real form E(R) of E. Let £^(E(R)) be the space of dis- 
tributions on the vector space E(M.) supported at the origin which is a convolution 
algebra with unit S, the Dirac delta function. Define a 'Fourier transform' T c by the 
following formula: 

(3.3) J- c :^ 0] (^(R))-Pol(E v ), Fcf(0'.= (f(x)Mx,0), (cE\ 
where $ is given by: 

Notice that we do not include v-l in the definition of <&(a;,£). This map is an 
algebra isomorphism and its inverse JTr 1 : Pol(i? v ) —> £^(E(R)) satisfies: 

T-\l) = 5. 

Remark 3.5. The algebraic Fourier transform defined in Definition 13. II satisfies 

(3.4) f = J" c oTojr-i f or T eV(E), 

and the formula (13. 4 p characterizes T because the Weyl algebra T>(E) acts faithfully 
on £LJE(R)) and so does V(E V ) on Pol(i? v ). The composition T c ° T a J 7 ^ 1 does 
not depend on the choice of a real form E(M). 
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3.2. Symbol map and reversing signatures. In the sequel we let E be a certain 
nilpotent Lie algebra n_. Even in the case when it is abelian, the left and right 
actions dL and dR cause different signatures. The purpose of this section is to set 
up carefully and clearly relations involving various signatures in connection with the 
algebraic Fourier transform. 
We define the symbol map 

Symb : Diff const (£)^Pol(£ v ), D x h> Q(Q 

by the following characterization 

(D x <S>)(xX) = Q((MxX)- 

The differential operator on E with symbol Q(() will be denoted by dQ x . 

For any homogeneous polynomial P on E v of degree I and any polynomial Q on 
E seen as a multiplication operator one has 

(3.5) dK = (-l) e P(C), Q0T) = dQ c . 

For further purposes notice that E z = - n, where E z := ZjTjf: stands for the 
Euler operator on the corresponding vector space. 

Denote 7 : S(E) — > Pol(.E v ) the canonical isomorphism, and define another algebra 
isomorphism 

lsgn :S(E)^Po\(EV), 

by 70 a, where a ■ S(E) -»■ S(E) is the automorphism induced by the antipodal map 
X h> -X for every X e E. 

Now we regard E as an abelian Lie algebra over C, and identify its enveloping al- 
gebra U(E) with the symmetric algebra S(E). Then, the right and left-infinitesimal 
actions induce two isomorphisms: 

dR : S(E) A Diff const (£), dL : S(E) ^ DiS const (E). 

By the definition of the symbol map, we get, 

Symb odR = 7, Symb odL = ^ sgn . 

On the other hand, it follows from (13.51) that 

dL(u) = -y(u), dR(u) = "fs 9 n(u), 

for every u e S(E) ^ U(E), where polynomials are regarded as multiplication oper- 
ators. Hence we have proved 

Lemma 3.6. For any u € U(E), one has: 

Symb odR(u) = dL(u), Symb odL(u) = dR(u). 
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3.3. Construction of equivariant differential operators by algebraic Fourier 
transform. In Section 12.31 we explained a general construction of equivariant dif- 
ferential operators between homogeneous vector bundles on different base spaces X 
and Y equipped with a morphism Y -*■ X by using the right action dR of U (g) on 
C°°(G'). Now we develop another construction based on the algebraic Fourier trans- 
form. In Theorem 13.91 we show that both methods give the same operators when the 
base space X is a flag variety G/P, where the parabolic subgroup P has an abelian 
nilradical. 

From now, let G be a real semisimple Lie group, P a parabolic subgroup of G with 
Levi decomposition P = LN + and V a finite dimensional representation of P. We 
apply the results of the previous sections to the case when H = P. 

Let LN_ be the opposite parabolic subgroup and n_(R) the Lie algebra of iV_. By 
the Bruhat decomposition we have the following open dense imbedding i : tl_(M.) 
G/P given by X h> exp(X)-o, where o = eP € G/P. The pullback of the (7-equivariant 
vector bundle V% -> G/P via i is trivialized into n_(M) x V w -*■ n_(R) and thus we 
have a linear isomorphism: 

(3-6) £' [o] (GlPX 2p ) — S[ 0] (n.(R),V v ), 

through which we induce the action of the Lie algebra g on £LJri-(M), V y ) denoted 
by dir. 

The Killing form of g identifies n^(IR) ^ n+(R) and thus the algebraic Fourier 
transform (13. 3 j) gives rise to a linear isomorphism: 

T c ® id : ^ 0] (n_(M), V) Pol(n + ) ® V\ 

through which we induce the action of the Lie algebra g further on the right-hand 
side, namely, 

(3.7) chc(X) :=F c odn(X)o T- 1 . 

By Remark 13.51 this action dir is given by the Fourier transform of operators: 

dn(X)=d7(X). 
In summary we have the following g-isomorphisms: 

(12.12b (I3.6li F c ®id 

(3.8) F c :ind«(F v ) ^ £[ o] (G/P, V 2 v p ) ^ 0] (n_(E), F v ) — Pol(n + ) ® 

Remark 3.7. (1) The map F c does not depend on the choice of a real form G of 
Gc that appears in the two middle terms of (13. 8p . This fact will be used for actual 
computations (see Section [6]). 

(2) The isomorphism F c : indp(y v ) — >■ Pol(n + ) <8> V w only depends on the infini- 
tesimal action of P on V, and so does dn. 
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Let E be a hyperbolic element of g defining a parabolic subalgebra p(E) = 1(E) + 
n(E), namely, 1(E) and n(E) are the sum of eigenspaces of a.d(E) with zero and 
positive eigenvalues, respectively. 

Let G' be a reductive subgroup of G and g' = Lie(G') ® C. 

Definition 3.8. A parabolic subalgebra p is said to be q'- compatible if there exists 
a hyperbolic element E' € q' such that p = p(E'). 

If p = 1 + n is g'-compatible, then p' := p n q' becomes a parabolic subalgebra of 0' 
with the following Levi decomposition: 

p' = r + n' :=([n g ') + (nn '), 

The key tool for the F-method that we explain in Section I3"31 is the following assertion 
that the two approaches (the canonical invariant pairing (12. lip ) and the algebraic 
Fourier transform (13.81) ) give rise to the same differential operators: 

Theorem 3.9. Suppose p is a parabolic subalgebra q which is g' -compatible. Assume 
further the nilradical n + is abelian. Then the following diagram commutes: 

F c ®id Symb (giid 

Honied, ind|J(K v )) Pol(n + ) 9 Hom c (V, W) ^- Diff const (n_) ® Rom c (V, W) 

u O u 

Rom p ,(W\mdl(Vv)) Diff G ,(Vx, Wy). 

Proof. Take an arbitrary (p e Hom P '(W /v ,indp(l /v )), which may be written as 

<p = £ Uj ® if)j 6 U(n-) ® Hom c ( V, W) 

j 

by the Poincare-Birkhoff-Witt theorem U(q) ^ U(x\-) <8> U(p). Then it follows from 
f[27L2l) and ([32]) that 

F c </? = Y^Fc dL( Uj )5 <2> e Pol(n + ) ® Hom c ( V, W). 

j 

Since 5 = < 7 r ~ 1 (l), we get 

F c (p = Y,dL(uj) ®ipj. 
j 

On the other hand, by the construction (12.81) . 

Dx^y(v) = Y^dRiuj)®^. 

j 

Now we use the assumption that n + or equivalently n is abelian. Then, in the 
coordinates n °> G/P the operator dR(uj) for Uj € £/(n_) defines a constant coeffi- 
cient differential operator on n_. Thus Dx^y(<p) can be regarded as an element of 
Diff const (n_)®Hom c (y,W). 
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where the last equation follows from Lemma 13.61 Thus we have proved that 



3.4. Fourier transform of principal series representations. As a preparation 
of the F-method, we recall some standard facts on principal series representations of 
complex reductive Lie groups. Let Pc = L C N + be a parabolic subgroup of a connected 
complex reductive Lie group Gc- Let A be a holomorphic representation of Lc on 
V, and extend it to Pc by letting the unipotent radical N+ act trivially. We form 
a Gc-equivariant holomorphic vector bundle V and V* = over the (generalized) 
flag variety Gc/Pc associated to A and \i := A v ® C2 P , respectively. We consider the 
representation ir^ of G c on C°° (Gc/Pc, V^)- We use the same notation ir^ for the 
representation of a given subgroup of Gc on C°°(U, V^Ju) if U is an invariant open 
subset of Gc/Pc- 

According to the Gelfand-Naimark decomposition g = n + I + n + of the complex 
reductive Lie algebra g, we have a diffeomorphism 



We use the same letter 7r M to denote the 'action' of Gc on G°°(n_, V v ), more precisely, 



for g € Gc and X e n such that g 1 exp X e G^ es . In particular, for g = mexp Z, with 
m 6 Lc, £ ti_, we have 



(F c <g> id)(p = (Symb ® id) o D x ^y(v) 



whence the Theorem. 



□ 



n_xL c xn + ^G c , (X,£,Y) » (expX)i(expY) 
into an open dense subset Gjf g of Gc- Let 

p ± : G£ g — n ±) Po : G^ - Lc, 
be the projections characterized by the identity 

exp(p-(s))po(s) exp(p + (^)) = 
For a section / e C°°(G C /Pc, V 2 v p ), we define F e G°°(n_, 1/ v ) by 

:=f(expX), for X e n_. 



(7rMF)(X)=fi( Po (g- 1 ex P X)r 1 F(p4g- 1 ex P X)) 



(3.9) 



(ix,(g)F) (X) =/ u(m)F(Ad(m) 



X-Z). 
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The goal of this subsection is to analyze the infinitesimal action dn^(Y) for X e q 
and its algebraic Fourier transform dn^Y), in particular for 7 e n + . For this, we 
introduce the following two maps: 



(3.10) a: fl xii_-[, (Y,X) » j f 

(3.11) /3:gxn_-n_, (Y,X)»£ 

dt 



t=o 



t=o 



Po{e tY e x ), 
p_ (e* y e x ) . 



We may regard f3(Y, ■ ) as a vector field on n_ through the following identification 
X » fi(Y,X) en^T x n_. 

Example 3.10. G c = GL(p + q, C), L c = GL(p, C) x GL(q, C), and n s M(p,q;C). 
We note that n is realized as upper block matrices. Then for g^ 1 = f ° e G c , 

Y = (c d) € M(yP + q] C ^ and X 6 M ( p ' q] C ^ we have 
p_{g- l )=bd-\ 

Po (g- 1 ) =(a - bd^c, d) e GL(p, C) x GL(q, C), 
a(Y, X) =(A - XC, CX + D)e gl p (C) + gl q (C), 
P(Y, X) =AX + B- XCX - XD. 

With the notation (13.101) and (13.111) . the infinitesimal action dn^ on C°°(n_,l /V ) 
is given by: 

(3.12) (drr fl (Y)F) (X) = »(a(Y, X))F(X) - (/3(Y, ■ )F)(X) for Y e flj 

where, by a little abuse of notations \x stands for the infinitesimal action as well. 

The right-hand side of (13.121) makes sense whenever /i is a representation of the 
Lie algebra I without assuming that it lifts to a representation of Lc, and thus we 
get a Lie algebra homomorphism 

(3.13) dvr / ,:g^P(n_)<2>End(y v ). 

If n + is abelian, we get another Lie algebra homomorphism by the algebraic Fourier 
transform on the Weyl algebra 2?(n_), see Definition 13.11 

(3.14) dn~:g^V(n + )®End(V v ). 

We pin down a concrete formula, which follows immediately from the definition 
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Proposition 3.11. Let (A, V) be an [-module, and set fx := A v <8>C2 P . We extend A v 
to a p-module by letting n + act trivially, and define dir^ by (13. 14ft . Then the algebraic 
Fourier transform of generalized Verma modules (see ( 13. 8\i ) 

F c : ind 5 p (V w ) Pol(n + ) ® V" 

intertwines the Q-action on the generalized Verma module indp(y v ) = U(q) ®t/(p) 
(A V ,F V ) with d^~. 

We begin by analyzing dir^ on the subalgebra (. Let L + : [ -> V(n + ) be the 
representation of ( by vector fields on n + given by 



Y »L + (Y) X := 4- 
v ' dt 



Ad(e- tY )x. 



Lemma 3.12. The following two representations of I on Pol(n + )®y v are isomorphic: 
(3.15) d7tJ| ( =:L + ®id + id®(^-2p)(y). 

Proof. For Y e [, X e n_ we have a(Y,X) = Y, and the formula f )3. 12D reduces, in 
2?(n_) ®End(V v ), to 

dn^Y) = id ® /i(T) - /?(Y, • ) ® id. 

We apply Lemma I'd. 31 to the case where (a, E) is the adjoint representation of 1 on 
tt_. With the notation therein we remark that \l/ a d = ~/3(Y, •). Then, we get 

dir~(Y) = id ® (i(Y) + ^ ad (Y) ® id + Trace o ad(Y) 

n_ 

= id®n(Y) + L + (Y) ®id-id®2p(y)id. 
Thus, Lemma follows. □ 

The differential operators dn^Y) with Y e n + play a central role in the F-method. 
We describe their structure by the following 

Proposition 3.13. For every Y e n + the operator dir^iY) is of degree -1 and more 
precisely it is of the form 

ft 2 ft 

where a{ and W are constants depending only on Y . 

Proof. Since n + is abelian, we can take a characteristic element H such that 

Kd{e tH )Y = e% foranyyen+. 
The equality (13.15P implies that 

(0 = A v (£)/i(Ad(r 1 )0, for^e n+, and^ := A v ® C 2p . 
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In particular, we have 7r^(e tH ) = a^ H ^R a , where a ■= e~ l and (R a h)(£) := h(a£). 
Then the identity 

Me tH )d^(Y)Me~ tH ) = ^{Y), 

implies 

RJ^{Y)R a l = a l d^{Y). 

Hence dn^Y) is of degree -1 for any Y e tl+. As dn fl (X) is a vector field there is no 
derivatives of higher order in the expression ( 13.161) . □ 

3.5. Recipe of the F-method. Our goal is to find an explicit form of a G'-inter- 
twining differential operator from Vx to Wy. Equivalently, what we call F-method 
yields an explicit element in Hom /(indp,(VF v ), ind£(V v )) ^ Hom p /(iy v ,ind^(y v )). 
Our assumption here is that p = ( + n+ is a g'-compatible parabolic subalgebra of g 
with abelian nilradical n + . In particular, p' := p n g' is a parabolic subalgebra of g' 
with a Levi decomposition p' = V + n' + where I' := In g' and := n + n g'. 
The method we develop is as follows: 

Step 0. Fix a finite dimensional representation (X,V) of p = [ + tt + . In case A lifts 
to a group Pc, we form a G^-equivariant holomorphic vector bundle Vx on 
X = Gel Pi£. 

Step 1. Consider a representation fi := A v ® C 2p of the Lie algebra p on V? '■= V y ® 
C 2p (- V v ), and compute (see fl3T3|) and (jSHJ) ), 

d7T M : -» P(n_) ®End(y v ), 
3tt~:0 -> P(n+) ®End(F v ). 

Step 2. Find a finite dimensional representation (y, W) of the Lie algebra p' such that 

Hom p ,(W/ v ,indS(y v )) * {0}. 

in case W lifts to a group we form a G^-equivariant holomorphic vector 
bundle W Y on Y = G'JP^. 
Step 3. Consider the system of partial differential equations for -0 e Pol(n + )®Hom(V, VF): 

(3.17) (d^(A)®id M /+id®z/(A))^ = for A el', 

(3.18) (rf^(G)®id w + id®z/(G))^ = forCen+. 

Notice that equations (I3.17P are of first order, whereas the equations (13 . 18[) 
are of second order. 

Step 4. Use invariant theory and reduce the system of differential equations (I3.17P 
and (13.181) to another system of differential equations on a lower dimensional 
space S. Solve it. 
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Step 5. Let ip be a polynomial solution to ( 13 . 1 T[) and (13.18P obtained in Step 4. 
Compute (Symb ® id) -1 (■?/;). This short-cut gives the desired equivariant 
differential operator in the coordinates n_ of X by Theorem 13.91 As a 
byproduct, (F c <g> id) _1 (-0) gives an explicit element in Hom P '(VF v , indp(V v )) 

(~ Hom /(indp,(H /v ), indp(y v ))), which is sometimes referred to as a singular 
vector. 

For actual applications in this work we assume in Step that dim V = 1 and that 
X is a Hermitian symmetric space G/K so that Vx is a holomorphic line bundle over 
X, and in Step 2 that (z^, IV) is irreducible so that n' + acts trivially on W. In this 
case the equation ( I3.18P is given as (dir^C) <8> id^)^ = 0, for which we shall simply 
write as di: ll {C)ip = 0. 

In Step 2 we can use explicit branching laws (see Fact 14.41 and Theorem I4.3|) to 
find all such W when (g(M), £}'(R)) is a reductive symmetric pair, n + is abelian, V 
is irreducible with a sufficiently positive parameter A. 

In Step 3, branching laws (Step 2) assures the existence of solutions to (I3.17P 
and ( 13.18)) . Conversely, these differential equations are useful in certain cases to 
get a finer structure of branching laws, e.g., to find the Jordan-Holder series of the 
restriction for exceptional parameters A (see [K0SSj ). 

In Step 4, we can take S to be one-dimensional in the case where G/G' is a 
reductive symmetric space of split rank one. 

3.6. F-method — supplement for vector valued cases. In order to deal with 
the general case where the target Wy is no longer a line bundle but a vector bundle, 
i.e., where W is an arbitrary finite dimensional, irreducible ['-module, we may find 
the condition (13. 1T|) somewhat complicated in practice, even though it is a system 
of differential equations of first order. In this section we give two useful lemmas to 
simplify Step 3 in the recipe by reducing ( 13 . 1 T[) to a simpler algebraic question on 
polynomial rings, so that we can focus on the crucial part consisting of a system of 
differential equations of second order ( 13. lSj) . The idea here is to work first on the 
highest weight variety of the fiber W, and will be used in Sections 16.21 and 16.31 

We fix a Borel subalgebra b(l') of V. Let x : b([') -> C be a character. For a 
completely reducible ['-module U, we set 

U x :={ueU : Zu = x(Z)u for any Z e b([')}. 

Notice that dim U x = 1 if and only if U is irreducible and with highest weight x- 

Let W be an irreducible representation of [' as before, and x f ne highest weight of 
the contragredient representation W y . We fix w w € (W w ) x a nonzero highest weight 
vector. Then we have the following: 
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Lemma 3.14. For an element tp e Pol(n + ) ® Hom(V, W), we set P := (tp,w y ) e 
Pol(n + ) ® V v . Then if) satisfies ( 13 . 1 7|) and (" 13 . 18[) if and only if the polynomial P 
belongs to (Pol(n+) <8> V w ) x and satisfies the equation (13.181) . 

Proof. Denote A the action of [ on V and v the action of [' on W. Owing to Lemma 
13.121 we have the identity: 

{ifj e Pol(n + ) ® Hom c (V, W) : tp satisfies pTfjl } = (Pol(n + ) ® Hom c (V r , V^)) 1 ', 

where I' acts on Pol(n + ) ® Hom c (y, W) by L + <g>id + id®(A v ®id + id®z/) on the right- 
hand side. 

For an ['-module U, we notice that the following contraction 

U ® W ® Cw v -> [/, u®w®w v ^(w,w v )u 

is a b(l')-homomorphism. Therefore, if {/ is completely reducible, the evaluation 
map 

u®w^u, ifj»(ij,w w }, 

induces a bijection between two subspaces: 

(3.19) {U®W) V U x . 

Applying fl3TT9l) to U := Pol(tt+) ® Hom c (F, W 7 ), we get Lemma EH □ 

Since any nonzero vector in W v is cyclic, the next lemma explains how to recover 
Dx^y if) from P given in Lemma [3.141 

We assume, for simplicity, that the [-module (A, V) lifts to L c , the ['-module 
(u, W) lifts to L' c , and use the same letters to denote their liftings. 

Lemma 3.15. For any if e Hom p ^(VF v , ind£(y v )) ; £ e L' c and w v e W v , 

(3.20) (DxMv), v v (i)w v ) = (Ad(£) ® A v (£)) (DxM^)^) . 

Proof. We write = ® V'j 6 ^( n -) ® Homc(V, W). Since y? is p'-invariant, we 
have the identity: 

Y, Uj ® ipj = Y, Ad(^)Wj ® z/(£) o ^ o A(r : ) for [ e L^. 

j j 

In turn, we have 

(DxMv)," v (tW) = ^^(Ad^KO^^^^oAcr 1 ) 

i 

= «Ad(e)®\v(t))(DxMv)^ v )- 

Thus, we have proved Lemma. □ 
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We notice that the right-hand side of (I3.20p can be computed by using the identity 
in Diff const (n_)®F v : 

<ZW (if) ,w v } = (Symb- 1 ® id y v ) (P) , 

once we know the polynomial P = (tp,w w ) with ip = (F c ® id)(^) (see Theorem 13. 91) . 
In Sections 16 . 2 1 and 1 6 . 3 1 we find explicit formulae for vector-bundle valued equivariant 
differential operators by solving equations for the polynomials P. 

4. F-METHOD FOR HERMITIAN SYMMETRIC SPACES 

From now we apply the general theory developed in the previous sections to the 
case where (G,C) is a reductive symmetric pair and where Dx^y intertwines holo- 
morphic discrete series representations. 

4.1. Branching laws. Our subject is to construct an explicit covariant differential 
operator from Vx to Wy. The existence, respectively the uniqueness (up to scaling) 
of such operators are subject to the conditions 

(4.1) dim Dift G ,(V x , Wy) > 1, respectively < 1. 

We then encounter a question to find the geometric settings (i.e. the pair Y c X of 
generalized flag varieties and two homogeneous vector bundles Vx X and Wy -*■ Y) 
that satisfies (14. ip . This is the main ingredient of Step 2 in the recipe of the F- 
method, and thanks to Theorem 12 .7\ the existence and uniqueness are equivalent 
to the following question of (abstract) branching laws: Given a p-module V, find 
all finite dimensional p'-modules W such that dimHom p '(VF v , indp(y v )) = 1, and 
equivalently, 

(4.2) dimHom ,(ind^(lF v ),ind9(y v )) = 1. 

This subsection briefly reviews what is known on this question (see Fact I4.2p . 

Let q be a complex semisimple Lie algebra, and j a Cartan subalgebra of q. We 
fix a positive root system A + = A + (g,j), write p for half the sum of positive roots, 
a v for the coroot for a e A, and g a for the root space. Define a Borel subalgebra 
b = j + n with the nilradical n := Qe A+ Qa- 

The BGG category O is defined as the full subcategory of jj-modules whose objects 
are finitely generated, j-semisimple and locally n-finite [BGG76 . 

As in the previous section, fix a standard parabolic subalgebra p with Levi decom- 
position p = [ + n + such that the Levi factor [ contains ). We set A + ([) := A + n A([, j). 
The parabolic BGG category O p is defined as the full subcategory of O whose objects 
are locally [-finite. 

The set of A e )* whose restrictions to j n [[, [] are dominant integral is denoted by 
A + ([) := {A € j* : (A, a v ) e N for any a e A + (l)}. 
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We write for the finite dimensional simple [-module with highest weight A, regard 
it as a p-module by letting n + act trivially, and consider the generalized Verma 
module 

ind«(A)=ind«(H):= U(g) ®u(p) V\- 
Then indp(A) € O p and any simple object in O p is the quotient of some generalized 
Verma module. If 

(4.3) (A,« v ) = for all aeA(Q, 

then V\ is one-dimensional, to be denoted also by Ca- In this case we say indp(A) is 
of scalar type. 

Let t e Aut(g) be an involutive automorphism of the Lie algebra g. We write 

g ±T := {v G g : TV = ±v} 

for the ±1 eigenspaces of r, respectively. We say that (g,g') is a symmetric pair if 
g' = g T for some r. 

For a general choice of r and p, the space considered in (14. 2p may be reduced to 
zero for all p'- modules W. Suppose V = V\ with A e A + ([) generic. Then a necessary 
and sufficient condition for the existence of W such that the left-hand side of (14.21) is 
nonzero is given by the geometric requirement on the generalized flag variety Gc/Pc, 
namely, the set G c Pc is closed in Gc, see |K12[ Proposition 3.8]. 

Consider now the case where the nilradical n + of p is abelian. Then, the following 
result holds : 

Fact 4.1 ( [K12] ). If the nilradical n + of p is abelian, then for any symmetric pair 
(g,g r ) the restriction of a generalized Verma module of scalar type indp(-A)| t ( g r) is 
multiplicity-free for any embedding f-Q T -+g such that i(G^)Pc is closed in Gc and 
for any sufficiently positive A. 

A combinatorial description of the branching law is given as follows. Suppose 
that p is g T -compatible (see Definition 13.81) . Then the involution r stabilizes [ and 
n+, respectively, the nilradical n + decomposes into a direct sum of eigenspaces n+ = 
n+ +n; T and G^Pc is closed in Gc- Fix a Cartan subalgebra j of I such that ) T := j ng T 
is a Cartan subalgebra of l T . 

We define G End(g) by 0|, = id and 6\ n++n - — id. Then 8 is an involution 
commuting with r. Moreover it is an automorphism if n + is abelian. The reductive 
subalgebra g Td = l T + n: T + n; T decomposes into simple or abelian ideals ©ig[ 9 , and 
we write the decomposition of n: r as n: T = ©jTr^ correspondingly. Each rr^ is a 
j r -module, and we denote by A(x\r T i , f) the set of weights of nz^ with respect to ) T . 
The roots a and f3 are said to be strongly orthogonal if neither a + ft nor a - (3 is a 
root. We take a maximal set of strongly orthogonal roots ^} in A(rr T i ,j T ) 

inductively as follows: 
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1) u[ % * is the highest root of A(rr r i ,j r ). 

2) is the highest root among the elements in A(nz T i , ) T ) that are strongly 
orthogonal to (1 < j < k{ - 1). 

We define the following subset of N k (k = £ fa) by 

(4.4) A + := fl A, Ai := {(af ) x ^ ki e N k > : a« > - > a£> > 0}. 

i 

Introduce the following positivity condition: 

(4.5) (\-p s ,a)>0 for any a e A(n+, j). 

Fact 4.2 ( |K08j ). Suppose p is q t - compatible, and X satisfies (14.31) and ( 14. 5ft . JTien 
i/ie generalized Verma module indjJ(-A) decomposes into a multiplicity-free direct sum 
of irreducible q t -modules: 

(4.6) ind*(-A)| r ^ ind£(-A| r - ££af^f). 

In particular, for a simple p T -module W (namely, a simple i T -module with trivial 
action ofn T ), 

dimHom fl r(ind^(W rV ),indg(C_ A )) = 1 

if and only if the highest weight of the \ T -module W is of the form X\^ + Y,i Ejti 
for some (a^- ) £ A + . 

In the latter part of the paper we shall construct a family of equivariant differential 
operators for all symmetric pairs (q,Q t ) with k = 1 (in particular, A(nZ T i ,) T ) is empty 
for all but one i). 

4.2. Local versus non-local intertwining operators for branching laws. Dis- 
crete branching laws for the restriction of a unitary representation of G to G' assure 
that there exist continuous G'-intertwining operators from the representation of G to 
irreducible summands of G'. In general such operators are integro-differential opera- 
tors, e.g., |KS] . In this section, however, we formulate and prove a quite remarkable 
phenomenon that any continuous G'-intertwining operators between two representa- 
tion spaces consisting of holomorphic sections is given simply by differential operators 
in certain settings, see Theorem 14.31 

Let G be a simple, connected, simply connected Lie group and K a maximal 
compact subgroup of G. We write c(£) for the center c(t) of the complexified 
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Lie algebra t := Lie(K) ® K C. Assume c(t) is nonzero. Then c(t) is actually one- 
dimensional, and there exists a characteristic element Z e c(t) such that the eigen- 
values of ad(Z) e End(g) is or ±1. We decompose 

q = I + n+ + n_ 

into the eigenspaces of ad(Z) with eigenvalues 0, 1, and -1, respectively. We note 
that I = t in this setting. Then p := [ + n + is a parabolic subalgebra with abelian 
nilradical n + , and the homogeneous G/K becomes a Hermitian symmetric space, for 
which the complex structure is induced from the Borel embedding 

G/KcG c /K c expn + = G c /P c . 

Furthermore, let r be an involutive automorphism of G. Without loss of generality 
we may and do assume that r commutes with the Cartan involution 9, and therefore 
induces a diffeomorphism of the symmetric space G/K. Thus, the subsymmetric 
space G T /K T c G/K is fixed pointwisely by r. We use the same letters r and 
9 to denote the complex linear extensions of their differentials. Since t9 = 9t, the 
involution r leaves the center c(t) of t invariant, and therefore one has either r\ t m = id 
or t| c ({) = - id because dimc(€) = 1. We say r is of holomorphic type if r| c ( t ) = id. In 
this case, the roles of 9 and r are the same as in the previous section, and we have: 

(a) the involution r acts holomorphically on the Hermitian symmetric space G/K; 

(b) the embedding G T /K T °> G/K is holomorphic; 

(c) p is g T -compatible. 

Here the complex structure on G T /K T is induced from the open embedding in the 
complex flag variety G^/P^: 

G T I K T ^ G/K 
open n n open 

Y= Cy PI - Gc/Pc =X. 

Notice that in this setting the complexified Lie algebra of K T coincides with the 
Levi part [ T of the parabolic subalgebra p T . Given a finite dimensional representation 
of K T , we endow the vector space 0(G T /K T , W) of global sections of the homoge- 
neous vector bundle W ->■ G T /K T with the Frechet topology of uniform convergence 
on compact sets. 

Theorem 4.3. Suppose the line bundle parameter A satisfies (14.51) . 
(1) Any continuous G T -homomorphism 

(4.7) 0(G/K, C x ) — ► 0(G T /K T , W) 



is given by a differential operator. 
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(2) Suppose W is irreducible as a K T -module. There exists a non-trivial contin- 
uous G T -homomorphism (14. 7p if and only if the K T -module W is of the form 
for some ae A + (see (14. 4p ). 
In this case, the homomorphism is given, up to a scalar, by the restriction 
to open subsets G T /K T c nl and G/K c n of the differential operator Dx^y 
constructed in Theorem \3.9l 

We denote by % 2 (M, V) the Hilbert space of square integrable holomorphic sec- 
tions of the Hermitian vector bundle V over a Hermitian manifold M. If (14. 5ft holds, 
then % 2 (GIK,C\) + {0}, and G acts unitarily and irreducibly on it. 

In order to prove Theorem 14.31 we recall the corresponding branching laws in the 
category of unitary representations, which are the dual of the formulas in Fact 14.21 In 
general G T is noncompact, and we need to consider infinite dimensional irreducible 
representations of G T when we consider the branching law G j G T . 

Given a = (ojj ) e A + (c N fc ), we write Wf for the G T -equivariant holomorphic 
vector bundle over G T /K T associated to the irreducible representation of l T with 
highest weight X\ r + £. E£i afvf. 

Fact 4.4 ( |KD5| ). If the positivity condition (03]) is satisfied, then H 2 (G T / K T 

is non-zero and G T acts on it as an irreducible unitary representation of G T for any 

a e A + . Moreover, the branching law for the restriction G j G T is given by 

(4.8) U 2 (G/K,£\) - X! ffi ^ 2 (C7^ T ,Wf) (Hilbert direct sum). 

aeA+ 

Proof of Theorem \4-3\ For continuous representations a, a' of G T , we write Homer (a, a') 
for the space of continuous G T -homomorphisms. 

Suppose cr is an arbitrary irreducible unitary representation of G T . Then for any 
irreducible A" r -module W, we have 

(4.9) dimRom G ,(a, 0(G T /K T ,W)) < 1. 

Moreover, suppose W\ and Wi are two irreducible iC T -modules. Then 
(4.10) 

dimRom G , (a, 0(G T I K T ,Wi)) * for % = 1, 2 => W x - W 2 as JT-modules. 

The assertion (I4.10p is an immediate consequence of the propagation theorem for 
multiplicity-freeness property under visible actions |K13 . 

In order to prove the first statement, it is sufficient to assume that W is asso- 
ciated to an irreducible i^ T -module W. Then it follows from (14. 9p . (I4.10p and the 
multiplicity-free branching law (JM]) that the space Hom G r (U 2 {G/K, £ x ) , 0(G T /K T , W)) 
is at most one-dimensional, and is nonzero if and only if there exists a € A + such that 
W - Wf . 
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0(R) 




g(R) re 


1 


su(n, 1) ©su(n, 1) 


su(n, 1) 


su(n, 1) 


2 


fip(n + l,M) 


sp(n,R) ©sp(l,R) 


u(l,n) 


3 


so(n, 2) 


so(n- 1,2) 


so(n- 1) ©so(l,2) 


4 


g) 


fi(u(l)©u(p-l,g)) 


s(u(l,g) ffiu(p-l)) 


5 


so(2,2n) 


u(l,n) 


u(l,n) 


6 


so*(2n) 


30(2) ©30^(272-2) 


u(l,rt - 1) 



TABLE 4.1. Split rank one irreducible symmetric pairs of holomorphic type 



Conversely, suppose that W ^ W^. Then, by the branching law for the generalized 
Verma modules (see Fact 14.21) . we have 

Honv (indf (W v ),ind£(C0)*{0}. 

Taking a non-zero element (p, we get a G T -equivariant holomorphic differential op- 
erator Dx^y(v) e Diff const (n_) ® Hom c (C A , PV) according to Theorem The 
restriction of Z^rO) gives an element of Hom G . (O (G/K, C\) , 0(G T /K T , Wf)). 
Since this space is at most one-dimensional, both statements of the theorem are now 
proved. □ 

In the sequel, by a little abuse of notation, we shall write X = G/K and Y = G T /K T 
and denote Dx~>y the intertwining differential operator given in Theorem 14.31 

4.3. Split rank one reductive symmetric pairs of holomorphic type. The 

homogeneous space G/G T is endowed with a G-invariant pseudo-Riemannian struc- 
ture g induced from the Killing form, and becomes an affine symmetric space with 
respect to the Levi-Civita connection. 

Definition 4.5. Geometrically, the split rank of the semisimple symmetric space 
G/G T is the dimension of a maximal flat, totally geodesic submanifold B in G/G T such 
that the restriction of g to B is positive definite. Algebraically, it is the dimension 
of the maximal abelian subspace of $j(R)~ r '~ 6 ' := {Y e g(R) : tY = 9Y = -Y}. We 
denote it by ranknjC/G 1 ". 

Assume from now that the split rank of the semisimple symmetric space G/G T is 
equal to 1, or equivalently, the real rank of g(M) r61 is one. Table |4~T1 above gives the 
infinitesimal classification of split rank one irreducible semisimple symmetric pairs 
(g(M),0(R) r ) of holomorphic type. 

The remaining part of this work is devoted to the explicit description of G T - 
intertwining operators between % 2 (X, C\) and its irreducible components given in 
Theorem 14.31 The next section treats all the cases where such operators are given by 
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normal derivatives. In Section [6] we analyze all the split rank one symmetric pairs of 
holomorphic type for which those operators have more intricate structure. 

5. Normal derivatives versus intertwining operators 



Suppose we are in the setting of Section |4.2[ namely, G T /K T is a subsymmetric 
space of the Hermitian symmetric space G/K. Then we can consider the Taylor 
expansion of any holomorphic function (section) on G/K along the normal direction 
by using the Borel embedding. This idea was used earlier by Martens, Jakobsen and 
Vergne |M75[ IJV79] for filtered modules to find abstract branching laws. 

However, it should be noted that normal derivatives do not always give rise to 
intertwining operators. In this section we clarify the reason and give a classification 
of all pairs (g(R), 0(M) T ) of split rank one for which it happens. 

5.1. Normal derivatives and the Borel embedding. Suppose E = E + + E _ is a 
direct sum of vector spaces. Then every u e E_ defines a vector field on E 

d 



f(v + tu), 



which is a normal derivative along the normal direction E_. 

More generally, we shall use vector-bundle valued operators T : C°°(E,V) -> 
C°°(E + ,W) given by normal derivatives. For this, we begin with the following set- 
ting. Let S(E-) be the space of symmetric tensors on E-, and suppose that W v is a 
subspace of S(E_)®V y . Let T denote the inclusion map T ■ W v S(E_)<S>V V . Then 
it determines an element, denoted by the same letter T, in S(E_) ® Homc(V, W), 
which can be viewed as an element of Diff const (i?) ® Homc(V, W) via the composition 

S(E.) c S(E) ^ Diff const (£). 

In the coordinates x = Y?j = i x j e j e where {ex, . . . ,e q } is a basis of it is 
written as 

T = Y, e a ®T a for some T a e Rom c (V, W). 

The subspace E + is given by the condition x = in E = {(x,y) ■ x e E_,y e E + }. 
Then the differential operator f : C°°(E,V) C°°(E + , W), f(x,y) » (ff)(y) is 
of the form 

/ > ,'d\ a \f(x,y) 
(5.1) ( ■>•>■>' A - ^ ' ' v 1 



{Tf)(y)= E T a (- 



dx c 



x=0J 



which is a normal derivative along the direction EL. 

We apply this construction to the subsymmetric space G T /K T in G/K. As in 
Section HJ let Vx be a homogeneous vector bundle over X = G/K associated to an 
irreducible representation V of K. Similarly, let Wy be a homogeneous vector bundle 
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over the subsymmetric space Y = G T /K T associated with an irreducible representa- 
tion W of K T . 

If there is a homomorphism T : W w — ► S(nz T ) ® V w then it defines a holomorphic 
normal derivative T with respect to n = nl + rr r , which induces a holomorphic 
differential operator between two vector bundles by 

(5.2) 0(n-,V) — >0(nZ,W) 



restriction 



restriction 



0(G/K,V x ) >0{G T IK^W Y ). 

We shall denote it by the same letter T, and call it a normal derivative along the 
direction n: T . 

5.2. When are normal derivatives intertwining operators? Let dim V = 1, 

and we write as before C\ for the homogeneous line bundle over X = G/K associated 
to the character Ca of K. 

Theorem 5.1. Suppose g(M) r ) is a split rank one irreducible symmetric pair 

of holomorphic type. Then, the following three conditions on the pair (g(R), g(IR) T ) 
are equivalent: 

(i) For any A satisfying (14.51) and for any irreducible K T -module W , all contin- 
uous G T -homomorphisms 

0(X,C x )^0(Y,W), 

are given by normal derivatives along the direction nz T . 

(ii) For some A satisfying (14.51) and for some irreducible K T -module W , there 
exists a non-trivial G T -intertwining operator 

0(X,£ x )^0(Y,W) 

which is given by normal derivatives of positive order. 

(iii) The symmetric pair (g(R), g(]R) T ) is isomorphic to one of (su(p, g),s(u(l) © 
u(p-l,g))), (so(2,2n),u(l,n)) or (so*(2n),so(2) ®so*(2n - 2)). 

Notice that the geometric nature of embeddings Y X mentioned in the condition 
(iii) corresponds to the following inclusions of flag varieties: 

P n C ^ Q 2n C; 
IGr^C 2 "" 2 ) - IGr„,(C 2 "), 
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where Gr p (C fe ) := {V c C k ■ dimV^ = p} is the complex Grassmanian, Q m C ■= {z e 
P m+1 C : ^o + '-' + ^m+i = °} is the complex quadric and IGr n (C 2n ) := {V cC k : dimV = 
n, Q\ v = 0} is the isotropic Grassmanian for C 2n equipped with a quadratic form Q. 

5.3. Outline of the proof of Theorem 15.11 The implication (i)=>(ii) is obvious. 
On the other hand, for split rank one symmetric spaces there are three other cases 
(i.e., (1), (2) and (3) in Table I4~T1) where the G r -intertwining operators are not given 
by normal derivatives. In Section O we construct them explicitly. This will conclude 
the implication (ii)=>(iii). For the rest of this section we shall give a proof for the 
implication (iii)=>(i). 

Consider a homomorphism: T : W y — ► S(nz T ) ® V v . We regard S(nz T ) ® V y as 
a subspace of Pol(n + ) ® V y on which the Lie algebra q acts by dn^, see (13.14=)) . If T 
is a K r -homomorphism, the differential operator T : 0(G/K,Vx) ~^ 0(G T /K T , Wy) 
is K T -equivariant . The following statement gives a sufficient condition for T to be 
G T -equivariant. 

Proposition 5.2. The normal derivative T induces a G T -equivariant differential 
operator from Vx to Wy if T is a K T -homomorphism and T(W V ) is contained in 

(Po\(n + )®V^ {<) . 

Proof. The proof is a direct consequence of the F-method as follows. First of all, we 
note that D e Diff cons (n_) <8> Homc(V, W) is a normal derivative along the normal 
direction nz T if and only if (Symb 8)id)(D) e Pol(n^ r ) ® Hom(V, W) in the diagram 
of Theorem 13.91 Furthermore, by Proposition 13.11) F c ® idw induces an isomorphism 
of two subspaces: 

(5.3) Hom p r(W v ,ind£(y v )) s (Pol(n + ) ® Hom(y, W))^^ . 

As V is the complexification of the Lie algebra of K T , T e Hom^r(VF v , ,S(rr T ) <8> V v ) 

defines an element in (Pol(n + ) ® Hom(y, W)) d ^^ { \ and therefore the invariance 
with respect to p T = l T + n+ reduces to that with respect to n+. 

Hence, by Theorem 13.9) the normal derivative T coincides with the G r -equivariant 
differential operator Dx->y(<£t), where ipr is the preimage of T in ( 15 .3)) . □ 

Lemma 5.3. Suppose (g(M),0(IR) T ) is a split rank one irreducible symmetric pair 
of holomorphic type and A satisfying ( 14. 5p . For a e N we define an V -module: 

(5.4) Wl := S a (n- + T ) ®C A . 

(1) The module W? is irreducible for any a e N. 

(2) If for an irreducible K T -module W there exists a nonzero continuous G T - 
homomorphism 0(G/K,C\) 0(G T /K T ,W) , then the module W is isomor- 
phic to for some a e N. 
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(3) Assume that 

(5.5) Hom r (5 a (nI T ),^ ai (n!) ® S a - ai (nZ T )) = {0} for any 1 < a x < a. 

Then, the normal derivative T corresponding to the natural inclusion T : 
(W?) w -> S^rr^) (Ca) v is a G T -equivariant differential operator. 

Proof. If T8aik^G/G T = 1, then the noncompact part of g(R) re is isomorphic to 
su(l,n) for some n. Thus the first statement follows from the observation that the 
action of [ r on n+ T corresponds to the natural action of gl n (C) on C n . 

The second statement is due to Theorem 14.31 for k = rank^ G/G T = 1. 

We have the following natural inclusions A c B => C, where 

A := Por(rC) ® CI, B := Pol a (n + ) ® C x , C := (Pol a (n + ) ® C v x )^\ 
Therefore 

Hom r ((iy A a ) v ,A) ^Hom r ((W / " A a ) v , J B) ^ Hom r ((W / " A a ) v , C). 

The left-hand side is an isomorphism because of the assumption (I5.5p . Moreover, 
since A is isomorphic to the irreducible [ r -module W x , the first term is one-dimensional 
by Schur's lemma. The last one is also one-dimensional according to the multiplicity- 
one decomposition given in Fact 14.21 Therefore, all three terms coincide. 

Hence the canonical isomorphism T : (W x ) v -> S^n: 1 ") <8> (C A ) V satisfies the as- 
sumption of Proposition 15.21 Thus Lemma follows. □ 

Remark 5.4. The highest weight vectors of the generalized Verma module indp(C A ) 
with respect to p T have a significantly simple form if the condition (15.51) is satisfied. 
In fact, by Poincare-Birkhoff-Witt theorem indp(C A ) is isomorphic, as an [-module, 
to S(xi-) ® C A , when n_ is abelian. Under the assumption (15.51) we thus have 

T oo 

(ind«(ci)) n+ .05>: r )®^. 

a=0 

This formula is an algebraic explanation of the fact that G r -equivariant operators 
are given by normal derivatives in this setting. 

In order to conclude the proof of Theorem 15.11 we have to show that in all cases 
mentioned in (iii) the condition (15.51) is fulfilled. It will be done in the next subsection. 

5.4. An application of the classical branching rules. In what follows, we shall 
verify the condition f !5.5j) for the last three cases (4), (5) and (6) in Table H~T1 by 
using some classical branching rules of irreducible representations of gl m (C). 

Denote by F(gl m (C),/i) the finite dimensional irreducible gl m (C)-module with 
highest weight fj,. For example, the natural representation of the Lie algebra gl m (C) 
on C m corresponds to F(gl m (C), (1,0, . . . ,0)) and its contragredient representation 
on (C m ) v to F(gI m (C), (0,0, . . . ,0, -1)), while the action of gl m (C) on the space 
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of symmetric matrices Sym(m,C) ^ S 2 (C' m ) given by C ^ XCX for X e gl m (C) 
and C € Sym(m,C) corresponds to F(g( m (C), (2, 0, . . . , 0)). More generally, the 
action of gi m (C) on the space of i-th symmetric tensors is no longer irreducible and 
decomposes as follows: 

S 4 (Sym(m,C)) s (S 2 (C m )) 
(5.6) x © F(gl m (C), (2i 1 ,2i 2 ,...,2i m )). 

«l>...>«m>0 

In turn, the classical Pieri's rule implies the following decomposition for the tensor 
product of such modules: 

il>...>«m>0, £i>2ii>...>f m >2i m , 

U+-+!m=! E™l(^r-2i r )=fc 

Remark 5.5. The summand of the form F(0[ m (C), (£, 0, . . . ,0)) occurs in the above 
formula if and only if i 2 = ••• = i m = 0, hence i± = i and £ - 2i = k. This remark will be 
used in Section 16.21 

Example 5.6. Let G = U(p,q), G T = U(l) x U(p - l,q) and l T = t T (R) ® R C =; 
g[ 1 (C) + +qL(C). Then, the decomposition n = nl © n: T as an V -module 

amounts to 

(C P Y s C 9 - (C 12 (C p_1 ) v h C) © (C_i H C h C 9 ). 

Where M stands for the outer tensor product representation. Since the V -module 
S a (nz T ) is isomorphic to the irreducible module C_ a H Ch S a (C q ), the space 

Honi[T ( S a (trl r ) , C_ a2 h S ai (C^ 1 ) v m (S ai (O) ® S a2 (O)) ) 

z's not reduced to zero if and only if ai = and a 2 = a. Therefore, the condition (I5.5P 
z's satisfied. 

Example 5.7. Let G = SO(2,2n), G T = 17(1, n) and V = gl^C) + flI n (C). Then the 
decomposition n_ = nl © rr r as an l T -module amounts to 

C_i h C 2n =; (C_i h C n ) © (C_! m (C n ) v ). 

Tntts, \ T -module S a (nz T ) is isomorphic to the irreducible module C_ a H (S' a (C n )) v ; 
whereas we have an isomorphism of V -modules: 

S ai {n T _) ® S a2 (nl r ) =; C_ ai _ a2 h (S ai (C n ) h S a2 (C n )). 

T/ras, ine space Hom[r(5 a (nz T ),S' ai (nl)®<S' a2 (rr' r )) z's not reduced to zero if and only 
if ai = and a 2 = a. Therefore, the condition ( 15. 5 j) is satisfied. 
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Example 5.8. Let G = SO*(2n), G T = SO*(2n-2) x SO{2) and l T = 0l n _i(C) + 
0l x (C). In this case, the decomposition n = nl © n: T as an V -module amounts to 

(Alt(C n " 1 ) v h 1) e ((C"- 1 ) v m C_i). 

JTiws the V -modules S a (nz T ) and S ai (nH) <8> S a2 (nz T ) are given as 

S a (nZ T ) ^((C" 1 )^^, 

S ai (rT) ® S a2 (nl r ) ^( J S ai (Alt(C n - 1 ) v ) ^((C"- 1 )^) h C_ a2 . 

Hence the compatibility condition for the action o/g[ 1 (C) c [ r implies (15 .5p . 

Thus we have verified the assumption (15.51) for all the three symmetric pairs 
(g(IR), 0(M)' r ) and have proved the implication (Hi) => (i) in Theorem 15. II by Lemma 
0(3). 

6. Partial differential operators built on Jacobi polynomials 

In this section we apply the F-method to the remaining cases of split rank one 
symmetric pairs of holomorphic type (see Table H~Tl (1), (2) and (3)). In all three 
cases, the families of equivariant differential operators we discover are built on special 
values of Jacobi polynomials in one variable. 

We shall give a detailed account of each step of the F-method according to the 
recipe (see Section 13. 5j) in the first case. For the remaining two cases, we skip similar 
computations and highlight new features, e.g. a trick that allows to describe vector- 
bundle valued differential operators. 

6.1. Case of SO(n, 2) j SO(n-l, 2). Let n > 2 and Q be a non-degenerate quadratic 
form on C n+2 defined by Q(w) = Wq + ••• + w 2 n - w^ +1 . Then the special orthogonal 
group Gq := SO(C n+2 ,Q) acts transitively on the isotropic cone 

H c := {w € C" +2 x {0} : Q(w)=0}, 

and also on the complex quadric 

Q n C := E c /C* c P n+1 C, 

which is a flag variety containing C n as an open dense subset (open Bruhat cell): 



\z\ 2 \z\ 2 ' 
1 - ■ L -^ : z :l + — 



(6.1) C"^Q"C, _ 

We write p ■ He -> Q n< C for the natural projection. 

The quadratic form Q is of signature (n, 2) when restricted to the real vector space 
E R := \/-le + YJjtl ^ej, where {ej : < j < n + 1} is the standard basis in C n+2 . Thus 
we have an isomorphism: 

SO{C n+ \ Q) n GL R (E R ) * SO(n, 2). 
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Let G be its identity component S0 (n, 2). Then the G-orbit through the origin in 
the open Bruhat cell C n (c Q n C) is still contained in C n , and is identified with the 
Lie ball X := {z e C n : |^| 2 + 1 - 2z*z > 0, < 1}. 

Let r be the involution of G acting by conjugation by diag(l, . . . , 1,-1, 1). It 
leaves G invariant, and we denote by G' the identity component of G T . The group 
G' = SO Q (n -1,2) acts on the subsymmetric domain 

Y:=Xn{z n = 0}. 

Note that X ~ S0 (n,2)/ SO(n) x SO{2) is the bounded Hermitian symmetric do- 
main of type IV in n ^ C n and Y ^ S0 (n-l,2)/SO(n-l) x SO (2) a subsymmetric 
space of complex codimension one. 

We construct explicit holomorphic differential operators that give the projectors 
onto irreducible summands in the branching laws with respect to the symmetric pair 
(S0 (n,2),S0 (n- 1,2)). 

By using the Gegenbauer polynomial Cf(x) (see Appendix IA.2j) . we set 

(6.2) C?( X ,y) :- xiCf (-|) - |(-l) V (a)r( ^ -^_ 2k + 1} W^- 

For instance, Cq(x,i/) = 1, C"(x,y) = 2ay, C^ix^y) - 2a(a + l)y 2 - ax, etc. Notice 
that, C"(x 2 ,y) is a homogeneous polynomial of x and y of degree I. 

Theorem 6.1. Suppose A € Z satisfies A > n - 1, and a e N. Then, any G T - 
intertwining operator from 0(G/K, £\) to 0(G T /K T ,£\ +a ) is given, up to a scalar, 
by the differential operator of degree a: 

\ n-l / f) \ 

(6.3) D X ^Y,a-=Ca 2 [-A C „-i,— j. 

Remark 6.2. 

(1) The same statement remains true for intertwining operators between the uni- 
tary representations % 2 {GI K,C\) and % 2 {G T j 'K T , C\+ a ). 

(2) For every A e C and a e N, D x ^y,a given by the same formula, is a G T - 
equivariant holomorphic differential operator between two homogeneous hxm- 
dles Cx^G/K and C x+a ^G T /K\ 

Remark 6.3. This result is a 'holomorphic version' of the conformally covariant op- 
erator considered by A. Juhl |J09] in the setting S n °> S n+1 , with equivariant actions 
of the pair of groups SO(n + 1, 1) c SO(n + 2, 1), respectively. For the case n = 1 we 
recover the celebrated Rankin-Cohen brackets. 

In order to prove Theorem 16.11 we apply the F-method (see Section f3 . 5 1) . We set 
H Q := Eo, n +i + -En+1,0, and o = CH Q . Then, the Lie algebra q = so(C n+2 , Q) is a sum 

= n_ + (a + m) + n + 
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of -1,0, and 1 eigenspaces of ad(H ), respectively. The stabilizer of the base point 
w a = [1 : : ••• : : 1] e Q n C is a parabolic subgroup Pc with Langlands decomposition 
P<c - MqAqNc, where Ac = exp a and iVc = expn+. 

As Step 1 of the F-method we define a standard basis of n+ ^ C n by 

Q '■= Ej t0 - Ej jTi+ i - E j - E n+ xj (1 < j < n), 

and similarly a standard basis of n_ C n by 

:= Ej )0 + Ej !n+ %- Eqj + E n+ ij (l<j<n), 

Let Z = § E?=i ^ e n and Y = £™ =1 e n + . Notice that exp(Z) • w = z e C n c 
Q n C via (16.11) . Moreover, with the notations of (13.101) and (13.111) . one has 

a(Y, Z) = -(z,y)H a modm; 

P(Y,Z) = (z^E.-^Qniz)^—, 

l i=i az i 

where (z,y) = z x y x + ■■■ + z n y n and Q n (z) := z\ + ■■■ + z\. 

For fx € C, we define a character /i : a = Ci7 G C by H a fi, and extend it to 
a character of p, denoted by the same letter \i : p -> C, or simply by C M , as the 
composition of the projection p ->• p/(m + n) and the above map \x ■ a -> C. 

Then the infinitesimal action dn^(Cj) is given by 

1 d 

(6.4) dn^Cj) = -fxzj - ZjE z + ifi^?) — , 
where E z := E"=i^^t- 

Lemma 6.4. For C e C n ^ n + and Z e C n ^ n one has, 

3^(Q) = A— + £ f — l<i<n, 

where fi = A v <g> C2 P = -A + n ; £^ := £™ =1 Cj^ - D c ^ s ^ e d'Alembertian differential 
operator of symbol Q n {z). 

Proof. The proof is straightforward from Definition 13. II and (16.41) . □ 

As for Step 2 we write Q = ((', £ n ) € C n with (' = (d, . . . , Cn-i) e C rt_1 and, according 
to Lemma l5\3l (2), we set for A e C 

(6.5) W£:=5 a (n; r )®C A . 

The vector space WjJ 1 is one-dimensional because rr r ^ C. We denote by v the action 
of V on Wf. In our setting where dim]/ = dimW 7 ^ = 1, Step 3 reduces to find 
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polynomials ip which are, according to (I3.17P and (j3.18p . m-invariant, homogeneous 
of degree a and satisfying 

3£[(<W = (a A + E( A _ I 0Dc j ^ = o, 1 < j < n - 1. 
Lemma 6.5. 

(1) For the irreducible l T -module W£ given in (16.51) the highest weight of (W^) v 
is given by x '■= A + a. 

(2) For the m-module Pol(n + ) ® y v ; the \-weight space for b([ r ) is given by 

(Pol(n + )®V v ) x .PoP[eQ(f)], 
where we identify Pol(n + ) ® y v Pol(n + ) as vector spaces. 

Proof. First notice that in the present setting i T = so(n - 1,C) +so(2,C). The 
decomposition n_ = nl © rr T as an [ T -module amounts to 

C n h C_i * (C"- 1 h C_i) ffi(Ci C_i). 

Since we have isomorphisms as P-modules: 

S a (n-)^ S ai (nI)®S a2 (nI T ) = ® 5 ai (C n ->C. a 

ai+a2=a ai=0 

and since the t r -module S a (nr T ) is isomorphic to the irreducible module C_ a ixiC, we 
obtain 

Hom r (5 a (n: r ),^ a (n_)) - ® Hom r (C B C_ ai S ai (C 1 " 1 ) B C_ ai ) ® (C B C_ a2 ) 

CLl+CL2 =a 

= © Hom B0(n _ 1) (C,^ 1 (C t - 1 )) 

ai+a2=a 
0<i<f 

□ 

Suppose n > 3. There exists a polynomial 

(6.6) € C-Span lt a - 2j : < j < 
of one variable t such that every ^ € Pol a [£^, Q(£')] can be written as 

(6.7) m = (T a g)(0--=Qn-i(C)h 

We note that for a more general g e C[t] the map T a g is a (multi- valued) mero- 
morphic function of £ 1} . . . , ( n . 




MM 

WGn-l(C')/ 
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To implement Step 4 of the F-method we say that a differential operator R on C n 
is T -saturated if there exists an operator S on C such that the following diagram 
commutes: 

C[t] — C(Cl,...,Cn) 

R 

C[*]— C(Cl,...,Cn). 

Such an operator S is unique (if exists), and we denote it by T\R. We allow R to 
have meromorphic coefficients. We note that 

(6.8) Tl(R 1 -R 2 ) = Tl(R 1 )-Tl(R 2 ), 

whenever it makes sense. 

We give several examples of saturated differential operators that we shall use later. 

Lemma 6.6. For every < j < n one has: 

(6-9) Tl{cjE c -Q n -i(a^) = 0, 

(6.10) W( a _i) Cn _£ c Jlj = o. 

Proof. The proof of both statements is straightforward from the definition of T a . □ 
Lemma 6.7. Let d t := i4 and A r «-i := ^ + — h Jot. One i/ien has: 

(1) Tl(E c ) = a-# t . 

(2) Tj(^l4) = (a-^)- 

(3) n(Q^E Cl ^) = (a-l)(a-# t ). 

(4) rJCC^Ac^x) = t^ t - a)(d t -n - a + 3). 

(5) TJ (Q^C) A C n-i) = (0 t - o)(t? t - n - a + 3). 

(6) rKg^cc)^)^- 2 ^ 2 -^)- 

( 7 ) ^Mb)=*t. 

(8) zfttf^) 

Proof. Notice first that the identity (1) is equivalent to (2) according to (16. 9p and that 
the identity (3) may be deduced from (1) or (2) by (16.1 Oft . Furthermore, identities 
(4) and (5) on the one hand and (6) and (8) on the other are equivalent according 
to the very definition of T-saturation as t = -, = = . 

J \/<2n-l(C') 

Thus, it would be enough to show only identities (1), (4), (7) and (8). We give 
a proof for the first statement, the remaining cases can be treated in a similar way. 
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Let 1 < j < n - 1 . Then 



= aQn-i(C0^ I - - -A=q^(OV f ^ 



V^-i(C')/ V<5n-i(C) " \V<5-i(C'). 



□ 



Thus Step 4 of the F-method in this setting is given as follows: 

Proposition 6.8. Let T a be as in (16. 7ft . 

(1) ylny polynomial ip on n + o/ degree a satisfying ( 13. 17ft is o/ t/ie /orm ^ = T a g 
/or some polynomial g(i) of one variable, belonging to W. 6\) . 

(2) T/ie polynomial = (T a g)(C) of n-variables satisfies the system of partial 
differential equations (13 . 18[) if and only if g{t) satisfies the following single 
ordinary differential equation: 

(6.11) ((1 - s 2 )^ 2 - (l + (2A - n + l)s 2 )ti s +a(a + 2\-n + l)s 2 ) #(-V-Ls) = 0, 



or equivalently, o(-v-Ts) is, up to a scalar, the Gegenbauer polynomial 

\ _ n—1 

Ca 2 (S). 

Proof. The first statement of Proposition follows from (16. 7p and Lemma 13.121 

Consider for any fixed 1 < j < n the differential operator - dn^Cj). Accord- 

ing to identities (1-6) in Lemma [6.71 we have following identities: 

l ad) 

T t( Qn-l(C') EC _d_^ = (a _^ )(a _^_ 1) + ((a _ W = (a _ 1)(a _^ )5 



(# t - a)(# t - n + 3 - a) + r 2 (tf 2 - ti t ). 
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Summing up these terms we get 

T\ / Qn-iCCO ^-^^j = ( 1 + f 2^2 _ ( X _ (2A - n + l)t 2 )<d t -a(a + 2X-n + l)t 2 . 



Thus Proposition follows after the change of variable t = -y -Is. □ 

Thus we have carried out the crucial part of the F-method. Let us complete the 
proof of Theorem 16.11 

Proof of Theorem \6.1\ The uniqueness of the intertwining operator (16. 3p follows from 
the multiplicity-freeness of the spectral decomposition in the unitary case (Theorem 

EM- 

The explicit form of the intertwining operator Dx^Y,a follows from Step 5 of the 
F-method, which is given by Proposition 16.81 □ 

6.2. Case of Sp(n,R) j Sp(n-1,R) x Sp(l,R). Let G = Sp(n,R) be the Lie group 
of all linear transformations of R 2n preserving the anti-symmetric bilinear form given 

by J n := q"J. Let r be the involution of G acting by conjugation by i2n-2,2, 

where I p>q •■= |^ ^ j. We consider the symmetric pair (G,G T ), where G T = Sp(n- 
1,R) xSp(l,R). 

Denote H n the Siegel upper half-plane {Z e Sym(n,C) : ImZ » 0}. It is a 
Hermitian symmetric domain of type CI in n_ ^ Sym(?7,,C). The Lie group G = 
Sp(n, R) acts biholomorphically on H n by 

g-Z = (aZ + b)(cZ + dy l for g = T b \ € G, Z 6 H n , 

where the isotropy subgroup K of the origin is isomorphic to U(n). 

We set X := H n and Y := H n -\ x H\. Thus Y ^ X is a complex (j r -equivariant 
sub manifold of X. 

Consider the following decomposition of the Lie algebra q - sp(n, C) = n_ + t + n+, 
\C ^ (-^'^'^) w itli B = l B and C = t C. Notice that in the present setting 

We write G for the universal covering of G, and K for its connected subgroup with 
Lie algebra t. 

For A g C, the map f ^ C, A ^> -A Trace A is a character of t, which we denote 
by Ca according to that we have chosen a realization of n + in the lower triangular 
matrices. The character Ca lifts to K and defines a G-equivariant holomorphic line 
bundle £\ over X = G/K ~ G/K. Notice that in the present case C 2p = C n+ i. 
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Then, according to branching law in Fact I4.4[ we have to deal with vector bundles 
rather than line bundles because there exists a nontrivial ^-intertwining operator 
Dx^y(p) from 0(X, C\) to 0(Y,Wy) only if dimV^ > 1 except for the trivial case 
Wy - £>\\y- In order to give a concrete model of such a representation space W, we 
define a module of ( T ^ g[ n _ 1 (C) + fll^C) by 

(6.12) Wl := S a (n; r )®(-ATrace n ) 

* {S a ((C^Y) 9 (-ATrace^)) m F(gi 1} (-A - a)e n ). 

As a vector space W£ is isomorphic to the space Pol a (rr T ) of homogeneous polyno- 
mials on nz T - C n_1 . Thus, if W is taken to be W£, then the intertwining differen- 
tial operator can be thought of as an element of C^^- <8> Pol a [i>i, . . . ,v n -i\, where 
(vi, . . . , v n -i) are standard coordinates on n: T ^ C n_1 and those on n_ ^ Sym(n, C). 

Theorem 6.9. Suppose A e Z satisfies A > n, and a e N. T/ien any G T -intertwining 
operator from 0(X,£\) to 0(Y,W%) is given, up to a scalar, by 
(6.13) 



I 8 2 8 \ 8 

D x ^ a :=c^[ £ ^v d ——^ E v,— Uc[— ]®poi*k 

where the polynomial C^i^x.y) was defined in (16.21) fry i/ie Gegenbauer polynomial. 
Remark 6.10. 

(1) The same statement remains true for intertwining operators between the uni- 
tary representations % 2 {X,C\) and V. 2 (Y, W^). 

(2) For any A e C and a e N, the operator Dx^y,a defined by ( 16 . 1 3[) is a G T - 
equivariant holomorphic differential operator between two homogeneous bun- 
dles C X ^X and W a x ^Y. 

In order to prove Theorem 16.91 we apply the F-method. Its Step 1 is given by 

Lemma 6.11. For C e Sym(n,C) ^ n + and Z e Sym(n,C) ^ n_ one /ias 

dm^C) = /iTrace(CZ) + X! E C/W^z^- — > 

i<j k,£ OZij 

8 1 ( 8 2 8 2 



where \i - A v ® C 2p = -A + n + 1 . 

Proof. We embed the group Sp(n, M) into C/(n, n) and apply the results of Example 
13.101 with p = q = n. Thus, the first statement follows from the formula (13.121) . 
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We consider a bilinear form 

n + x n_ C, (C, Z) ^ Trace(C*Z), 

where n + ^ Sym(n, C) ^ n . Recall that ^ with 1 < z < j < n are the coordinates on 
Sym(?7,,C). However, it is convenient for the computations below to allow writing 

1 , , d T 



J- to denote Then 



" i K ' "' J/ <K,/ dz i3 



Thus the algebraic Fourier transform of the first term of dn^C) amounts to 

(Trace(CZ))~= Wc^ + h) A = £C y A, 
whereas that of the second term of dir^C) amounts to 



d \ d 1 <9 2 

E E CktZikZjt-^— =-(w + i)£ c^^t- -7 E CfcK 1 + + ^OCr 



d 1 ( d 2 d 2 

= -( n + 1 )E C 'ii^7 _ " 9 E C ki(ij-K7 Rr + E C kiCij R , o> 

Hence the formula for dn^C) follows. □ 

The Step 2 of the F-method is given by the branching law in Fact 14.21 and leads 
us to ( 16.1 2p . For Step 3 we apply Lemma 13.141 and we get: 

Lemma 6.12. Let be the irreducible [ T -module defined in (I6.12p . 

(1) The highest weight of (W") v is given by 

X=(o,0,...,0;o) + (A,...,A;A). 

(2) For the [-module Pol(n + ) ® the x-weight space for b([ T ) is given by: 

(6-14) (Pol(n + )®Q) x - Cd^nCL 

2j+k=a 

where we identify Pol(n + ) ® with Pol(n + ) as vector spaces. 

Proof. The statement (1) is clear from the definition of W£ given in (I6.12p . 

Notice then that in our convention A(n_) = {e« + ej] 1 < % < j < n}. Thus u_ 
decomposes into irreducible representations of l T as 

n_ * (Sym(n-l),C)HC)e(CHC 2 )e(C"- 1 HCi) 
(6.15) * (F( U,2 ei ) bi F^O)) (F(flU,0) m F(gl 1; 2e n )) 

®( i71 (s^-i 1 ei)HF(gl 1 ,e„)). 
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Accordingly we get an isomorphism of [ r -modules: 

(6.16) Pol(n + ) * S(n-) = © (S l (Sym(n - 1), C)) ® ^(C 71 " 1 )) a C 2j+k . 

i,j,k 

Since CiiiCnn an d (in are highest weight vectors in the l T -module n with respect 
to A + ([ T ) (see (I6.15P ). so is any monomial CuCnnCin m the [ T -module 5(n_) - Pol(n+) 
of weight (2i + k)ei + (k + 2j)e n . 

According to the irreducible decomposition ( 16 . 1 6[) and Remark I5.5[ it follows that 
the right-hand side of ( 16.141) exhausts all highest weight vectors in Pol(n + ) of weight 
a(ei + e n ). Thus, taking into account the T-action on ^ ATrace n , we get Lemma. 

□ 

As Step 4, we reduce the system of differential equations ( 13. 181) . i.e. dii ^(C)^ = 0, 
to an ordinary differential equation as follows: 

Proposition 6.13. 

(1) Any polynomial ift(() = ip(dj) in the right-hand side of l[6.14\ ) ^ s given by 

(6.17) V(C) = (Tag) (C) := (V^UTg f-=fe 



where g(t) is a polynomial in one variable t of degree at most a. 
(2) The polynomial ift(() of ^n(n + 1) variables satisfies the system of partial 
differential equations dir^C)^ = for any X e n+ if and only if g(t) satisfies 
the Gegenbauer differential equation 

(6.18) ((1 - t 2 )d 2 t - (1 + 2(n - fi)t 2 )$ t + a(a + 2(n - /i))t 2 ) g(t) = 0, 

where we denote $ t = t4r as before. 

Proof. The first statement is clear from (16.141) . The proof of the second assertion is 
similar to the one of Proposition 16.81 and uses the following identities: 

T^ u =T^ nn = l(a-# t ), T^ ln =A, 
where % = C^- □ 
We are ready to complete the proof of Theorem 16.91 

Proof of Theorem \6.S\ The existence and uniqueness of the continuous G T -intertwining 
operator are guaranteed by Theorem 14.31 Let us prove that Dx^y,a defined in 
(16.131) belongs to Differ (£ A ,W"). Using the F-method we have proved that if 
D e Differ (£ A , W") and w v is a highest weight vector in (W^) v , then (D,w w ) is 
of the form (Symb -1 ® id)T a g, where g(t) is a polynomial satisfying (I6.18p . Hence 
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g(t) is, up to a scalar multiple, the Gegenbauer polynomial C^ 1 (t). In turn, 
(T a g)(Q = C^ 1 (2CnCnn,Cin) up to a scalar. 

Thus, in order to show Dx~*Ya e ^iSa^(C\, W") it is sufficient to verify for all 

(6.19) (Symb ® id)(ZW,a, v*(t l )w<) = (Ad(t l ) ® X"(t l ))(T a g), 

by Lemma 13.151 and by the observation that every nonzero w v € W w is cyclic. The 
left-hand side of (16.191) amounts to 

[c*r x ( Y zmtaCnn, Y ^cA^c^k) 

\ \l<i,i<n-l l<j<n-l / / 

= (det£)" A lc*T l ( Y iWSvUijCnn, Y WjCjn),wA, 
\ \l<i,j<n-l l<j<n-l I I 

where v = %vi, . . . , v n -±) stands for the column vector. Since (Q(v),w v ) gives the 
coefficients of in the polynomial Q(v), it is equal to 

(deU)- A C a W Y MiltjlCijCnn, Y ^Cjn) 
\l<i,j<n-l l<j<n-l / 

= (det^-^f Y 2(*C0uU, E (*C)i»)- 

\l<ij<n-l l<?'<n-l / 

On the other hand, the action of Ad(£ _1 ) on Pol(n + ) is generated by 

Hence, the right-hand side of H6 . 191) amounts to 

(det£)- x C*- l ( Y E (%C)m), 

\l<ij<n-l l<j'<n-l / 

which concludes the proof of Theorem 16.91 □ 

6.3. Case of U(n, 1) xU(n, 1) j (rz, 1). Let 1) be the Lie group of all matrices 
preserving the standard Hermitian form given by I n> \. Consider the Lie group G := 
U(n, 1) x U(n, 1) and let r be the involution of G acting by r ■ (g, h) -*■ (h,g). We 
consider the symmetric pair (G,G T ), where G T A(U(n, 1)) c G. 

Let D be the unit ball {Z e C n ■ \\Z\\ < 1}. It is a Hermitian symmetric domain 
of type AIII in C n . We adapt the notations of Example 13.101 with p = n and q = 1. 
Then the Lie group U(n, 1) acts biholomorphically on .D by 

0-Z = (aZ + &)(cZ + tO _1 for € U(n, 1), Z e D, 

and the isotropy subgroup K at the origin is isomorphic to U{n) x £/(!). 
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We set X := D x D and Y := A(Y). Thus, we have the following diagram: 

X = Dx D c C n x C n k n c P'C x P»C 

u u u u 

y = A(L>) c A(O) s nl c A(P n C). 

For Ai,A 2 e C, the map fl( n (C) + gl^C) -» C, (A,d) ^ -Ax Trace A - X 2 d is a 
character, which we denote by C(a 1i a 2 ) according to that we have chosen a realization 
of n + in the lower triangular matrices. For integral values of Ai and A2 the character 
C (Ai ,a 2 ) lifts to U{n) x U{1). 

According to the branching law in Fact l4.4[ there exists a nontrivial G T -intertwining 
operator D x ^y(v) from 0(X, £(a' a 2 ) ®£(a;',a")) to 0(Y,Wy) only if dimV^ > 1 ex- 
cept for the case Wy - (£(a^,a' 2 ) ® ^(A'/jAJ,'))!^- fr 1 or der to give a concrete model of 
such a representation space W, we define a module of l T 2 flt n (C) + g^C) by 

(6.20) W? XiM) := 5 a (nr)®C (All A 2 ) 

* (S a ((C n ) v ) (-Ax Trace n )) H F( [ 1 , (-A 2 + a)e n+1 ). 

As a vector space Wg is isomorphic to the space Pol a (rr T ) of homogeneous polynomi- 
als on rr T ^ C n . Thus, if W is taken to be W?, , x, then the intertwining differential 
operator can be thought of as an element of 

(6.21) C 9 8 8 8 



.Pol a [>i,...,v|, 



dz[ ' ' dz' n ' cfe" ' ' dz" 

where (v±, . . . ,v n ) are the standard coordinates on rr T 2 C n and z'^z'- are those on 
n_^C n + C n . 

Let (t) be the Jacobi polynomial defined by 

p _ T(a + e + l) ' U\ Tja + P + i + m + l) 
£ U " r(a + + £ + 1)^ \mj £!r(a + m + l) V 2 I ' 

see Appendix for more details. We set 

P^\x,y) :=^(2j + l). 

For instance, Pq'^(x, y) = 1, (x, y) = (2 + a + + (a + l)y, etc. 

Theorem 6.14. Suppose that X[, X' 2 , A", X 2 € Z satisfy X' ■= X[ - X' 2 > and A" := 
X"-X 2 > 0, and that a e N. T/ien any G T -intertwining operator from 2 (Y, £(x' ,A'))® 

2 (Y,£(a» A 2 ')) to ^ 2 (^^ , (A'+A",A'+A 2 ')) ^ S ^ en ' U P to a sca ^ ar ; ^ 



(6.22) ZW, a := P ( 



-A'+n,A'+A"-2n-2a+l 



<9z,- ' hi " J dz 
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Remark 6.15. 

(1) The fiber of the vector bundle W? Ai A ^ is isomorphic to the space S a (C n ) 
of symmetric tensors of degree a. It is a line bundle if and only if a = 
or n = 1. In the case n = 1, the formula ( 16 .221) is nothing but the classical 
Rankin-Cohen bidifferential operator (we note that SU(1,1) is isomorphic 
to SX(2,R)). 

(2) D x ^y,a belongs to (I6.2ip . and is thus regarded as a W" A , +A „ A , +A ,^- valued 
differential operator (see Diagram 15. 2p . 

(3) The same statement remains true for the unitary representations. 

(4) For any A^, X r 2 , A", X' 2 ' e C and aeN, the operator Dx^Y,a defined by the same 
formula, is a U(n, l)-equivariant holomorphic differential operator between 
two homogeneous bundles {^(x'^x^)) ® ^(\'{,x^))) -+ X and VV" Ai x ^ -> Y. 

In order to prove Theorem 16.141 we apply again the F-method. Its Step 1 is given 

by 

Lemma 6.16. Let C := C + C" = (c£, . . . , + (c'/, ...,<') 6 tl+ =; C" © C n , 

i=i i=i 
d* M (C) e <i V ;.,;(C") = - (w - n' 2 + n + 1) £ cj— + £ 

The Step 2 of the F-method is given by the branching law in Fact 14.21 and leads 
us to (16.201) . For Step 3 we apply Lemma 13. 141 and we get: 

Lemma 6.17. For Ai, A2 e C and a e N we recall from (16.201) £/ia£ A ^ an 
irreducible module of \ T ~ 0t n (C) + 0l x (C). 

(1) T/ie highest weight of (W" Xi X A is given by 

X= (o,0,-,0;-a) + (Ai, Ai; A 2 ). 

(2) For the [-module Pol(n + ) <8> (C>,) v , the x-weight space with respect to b(l T ) is 
given by 

(6.23) (Poi(n + )®<q) x . e c(co j (cry, 

t+j=a 

where we identify Pol(n + ) ® C A with Pol(n + ) as vector spaces. 
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Proof. According to the natural decomposition n + ^ C n © C n , one has 
Pol(n+) * ®Pol l (C n )S)PoF(C n ) 

- ©©F(g[ n (C),( Sl ,-, Sn ))®F( [ 1 (C),-(z + j> n ), 

i,j S. 

where the sum in the second factor is taken over all s = (si,---,s n ) e N n satisfying 
si > ■■■>s n > 0, max(z,j) > s\ > min(z,j) and si + --- + s n = i + j. In particular, the y- 
weight space for b(l r ) occurs in Pol(n + )<8>C^ if and only if i+j = a and S2 = ••• = s n - 0. 
In this case the weight vectors are the monomials (CiYiCiV ■ Lemma follows. □ 

As Step 4, we reduce the system of differential equations (I3.18p . i.e. dn^C)^ = 0, 
to an ordinary differential equation as follows: 

Proposition 6.18. 

(1) Any polynomial if)(('X") of 2n-variables in the right-hand side of (16. 23|) is 
given by 

Tp(CX") = (T a g)(CX") ■■= (CiTs(|), 

where g(t) is a polynomial in one variable t of degree at most a. 

(2) The polynomial ifj(C'X") satisfies d7r(X)tp = for any X e n+ if and only if 
the polynomial g(t) solves the single equation 

(6.24) ((1 - , 2 )^ + q(s)± + a(l - a 2 - a)) g (^) = 0, 

where a% = fj,[ - fi' 2 + n + 1, 02 = - /4' + n + 1 and q(s) = s(a2 - 2 + 2a) - 1a\ - 
a 2 -2a + 2. 

For the proof of Proposition 16.181 we state following identities for T a -saturated 
operators whose verification is similar to one for Lemma 16.71 

Lemma 6.19. One has: 

(2) ^(^4),)=^. 

(3) n(a'^)=a-f t . 

(4) Tl ((CD 2 ^) = <a - 1) - 2(a - l)t| + t^. 

Proof of Proposition \6.18\ . The general conditions (13.1 7\i and (13. 18j) of the F-method 
reduce in the present case to one non-trivial differential equation that holds for 
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C 1 = (l,0,...,0) + (l,0,...,0)en- 
(6.25) 

(w - A + n + 1) A + + M - ,» + „ + d A + c;A_) W ,C) = . 



According to Lemma 16.191 the above requirement corresponds to the following con- 
dition on g(t): 



(6.26) 



d 2 



d 



(t + t 2 )-— + (ax - t(a 2 + 2a-2))— + a(a 2 + a- l)g(t) = 



dt 2 



dt 



where a± = - /4 + n + 1 and a 2 = /// - + n + 1. 

To complete the proof, notice that the change of variable t*-> |(s - 1) transforms 
the above equation into (I6.24p . □ 



Proof of Theorem \6.14\ Now we have all the necessary ingredients to apply Lemma 
13.151 The remaining part of the proof is parallel to one for Theorem 16.91 □ 

Remark 6.20. In all the three cases we have reduced a system of partial differen- 
tial equations to a single ordinary differential equation in Step 4 of the F-method. 
The latter equation has regular singularities at t = ±1 and 00. We describe the 
corresponding singularities via the map T a as below: 



(1) The singularities of the differential equation ( 16. lip correspond to the varieties 
given by ( n = and Q n -i(C) = 0. 

(2) The singularities of the differential equation ( I6.18P correspond to the varieties 

Cll Cln 



given by Cin = and det 



Cln Cn 



0. 



(3) The singularities of the differential equation ( I6.24p correspond to the varieties 
given by C( = and £{ = ±C"- 



Appendix A. Jacobi polynomials and Gegenbauer polynomials 

A.l. Jacobi polynomials. Let a > -1 and /3 > -1. Jacobi polynomials P^ ,l3 (t) of 
one variable t and of degree t form an orthogonal basis in L 2 ([-l, 1], (l-t) a (l+t)^dt) 
and satisfy the Jacobi differential equation 

/ d 2 d \ 

(A.l) ^i-t 2 )— + (p-a-(a + P + 2)t)- + e(t + a + p + l))y = 0. 
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This equation is a particular case of the Gauss hypergeometric equation. Therefore 

r(q+l+l) 

r(a+i)e\ 



its unique polynomial solution y(t) satisfying y(l) = t^THF * s given by: 



T(a + £+l) ' U\ T(a + f3 + £ + m + l) ( t-l Y 



T(a + (3 + £ + l) ~o\ m / T(a + m + l)e\ 
The Jacobi polynomials are subject to the Rodrigues formula 



(i - tni + ty P ^{t) = -^f (|) ((i - + o^) ■ 

When a = j3 these polynomials yield Gegenbauer polynomials (see the next paragraph 
for more details) and they reduce to Legendre polynomials in the case when a = /3 = 0. 
Here are the first four Jacobi polynomials. 

• Po'Ht) = 1. 

• P? p (t) = ±(a - /? + (2 + a + P)t). 

• P^(t) = i(l+a)(2+a) + i(2 + a)(3 + a+^)(t-l) + |(3+a+/3)(4+a+/?)(t-l) 2 . 

A. 2. Gegenbauer Polynomials. The Gegenbauer (or ultraspherical) polynomial 
Cg(t) is a polynomial of t of degree £ satisfying the Gegenbauer differential equation 

(A.2) ((1 - t 2 )d 2 - (1 + 2at 2 )$ t + £{£ + 2a)t 2 ) y = Q. 

It is a specialization of the Jacobi polynomial 

, , , x T(a + ^)T(£ + 2a) a .i a .i 

a. 3 c?m = -4 — r 1 ^ — 7-7 p» 21 2 m. 

v ; £ W r(2a)r(^ + a + i) 1 v y 

This equation (1A.2|) is a particular case of the Gauss hypergeometric differential 
equation. Thus 



( -1 \k A: + Ql) fr>4.\t—2k 

= j^" } T(a)T{k + l)T{£-2k + l) { } ' 
Here are the first five Gegenbauer polynomials. 

• q?(f) = 1. 

• Cf (t) = 2at. 

• cf(t) = -a(l-2(a + l)t 2 ). 

• C^(t) = -2a(a + - f (a + 2)t 3 ). 

• cf(t) = \a(a + 1)(1 - 4(a + 2)t 2 + f (a + 2) (a + 3)t 4 ). 
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